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ABSTRACT 


This  report  (Volumes  I  and  II)  documents  the  successful  completion  of  Phase  II  of  the  Defense 
Advanced  Research  Projects  Agency  (DARPA)  Improving  Warfighter  Information  Intake  Under 
Stress  (IWIIUS)  program.  Volume  I  discusses  the  Phase  II  integration  of  cognitive  state  gauges  into 
operational  systems  that  could  demonstrate  the  viability  of  augmenting  cognition  into  an  operational 
context.  Volume  II  (Appendices)  describes  the  successful  results  of  four  industry  teams  building 
Closed-Loop  Integrated  Prototype  (CLIP)  systems  that  demonstrate  how  the  limitations  of  human 
cognition  can  be  addressed  by  augmenting  cognition  with  advanced  cognitive  state  sensors  that 
provide  input  to  complex  computational  systems.  A  cognitive  bottleneck  was  identified  by  each  of 
the  four  development  teams  along  with  empirically  testable  goals.  The  bottlenecks  were  operationally 
defined  in  terms  of  the  application  environments  addressed  by  each  of  the  four  teams. 

The  Honeywell "  research  team  addressed  Attention  as  their  primary  bottleneck.  Attention  was 
operationally  defined  in  the  context  of  a  ground  infantry  soldier  in  the  U.S.  Army’s  Future  Force 
Warrior  (FFW)  program.  The  FFW  will  have  more  information  available  (via  physical  sensors, 
communications,  etc.)  and  processing  that  information  in  a  dynamic  and  lethal  environment  requires 
careful  management  of  limited  attentional  resources.  During  their  Concept  Validation  Experiment 
(CVE)  they  demonstrated  a  380%  performance  improvement  where  attention  resources  were 
required,  exceeding  the  100%  improvement  goal  set  by  DARPA.  Honeywell1'  was  able  to  correctly 
classify  attention  state  changes  more  than  98%  of  the  time,  in  less  than  300  ms,  again  exceeding  the 
Phase  II  operational  goals.  Finally,  they  were  able  to  change  mitigations  in  well  under  the  1 -minute 
deadline  established  by  DARPA  as  adequate  for  operational  testing. 

DaimlerChrysler  Corporation  (DCC)  met  all  their  bottleneck  performance  goals.  They  addressed 
the  Sensory  Input  bottleneck  that  had  significant  cognitive  impact  in  the  context  of  a  Light  Armored 
Vehicle  (LAV)  operator.  A  future  LAV  operator  will  have  multiple  cognitive  tasks  (e.g.,  communica¬ 
tion,  planning,  command  and  control)  to  manage.  Sensory  input  resources  are  required  by  these 
cognitive  tasks  as  well  as  by  the  primary  job  of  maneuvering  the  vehicle.  DCC  investigated  the 
sensory  bottleneck  in  the  auditory  and  visual  modalities,  with  multiple  mitigation  strategies.  Their 
experimental  results  under  real-world  driving  conditions  showed  that  the  sensory  bottlenecks  could 
be  improved  by  as  much  as  108%  with  an  accuracy  of  up  to  98%,  depending  on  the  modality  being 
examined.  The  sensory  bottleneck  status  could  be  detected  in  as  little  as  200  ms,  and  mitigations 
could  be  invoked  in  as  little  as  0.2  sec,  depending  on  the  mitigation  used.  The  DCC  team  met  all 
DARPA-defined  performance  criteria  for  Phase  II  of  the  IWIIUS  program. 

The  Lockheed  Martin8  Advanced  Technology  Laboratory  (LMATL)  addressed  Working  Memory 
as  their  primary  bottleneck  within  the  Tactical  Tomahawk  Weapons  Control  System  (TTWCS) 
simulation  environment.  They  developed  their  CLIP  for  a  Tactical  Strike  Coordinator  (TSC),  who 
must  manage  a  number  of  missiles,  targets,  and  shipboard  launch  platforms,  and  dynamically 
reassign  the  missiles  to  targets  as  critical  targets  pop-up,  missiles  fail,  etc.  The  operator  must  recall 
and  recognize  far  more  information  than  can  be  maintained  in  working  memory  to  perform  the  task 
effectively.  They  demonstrated  an  improvement  in  working  memory  throughput  by  at  least  500% 
in  the  context  of  a  U.S.  Navy  command  and  control  task.  LMATL  used  an  Intelligent  Sequencing 
mitigation  strategy  to  strategically  present  related  information  about  specific  missile-target  pairings 
when  the  Working  Memory  bottleneck  was  saturated,  achieving  a  642%  performance  improvement, 
well  in  excess  of  the  Phase  II  goal  for  this  bottleneck.  Working  memory  status  (high  or  low  load) 
was  correctly  identified  in  over  90%  of  the  trials.  The  sequencing  mitigation  took  place  in  less  than 
500  ms,  again  meeting  all  Phase  II  development  objectives  demonstrating  a  successful  mitigation 
for  the  Working  Memory  bottleneck. 
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Finally,  the  Boeing®  team  was  assigned  the  Executive  Function  bottleneck  in  the  context  of  an 
Unmanned  Air  Vehicle  (UAV)  systems  operator.  Executive  Function  is  analogous  to  the  central 
processor  in  a  computer,  and  relates  to  the  need  to  process  and  rapidly  assimilate  many  information 
elements  for  a  single  operator  to  manage  multiple  UAVs.  Having  one  operator  for  multiple  vehicles 
is  a  pivotal  goal  for  the  next-generation  U.S.  Air  Force  and  Joint  Unmanned  Combat  Air  System 
(JUCAS).  Boeing®  developed  an  array  of  mitigations  that  were  triggered  to  address  the  cognitive 
bottlenecks.  With  regard  to  their  targeted  bottleneck,  Executive  Function,  they  empirically  showed 
a  241%  performance  improvement,  with  a  classification  accuracy  of  92%  in  less  than  1  second. 

The  mitigations  were  determined  and  triggered  in  less  than  1  second  as  well.  Boeing®  met  all  the 
criteria  for  Phase  II  success,  not  only  for  the  Executive  Function  bottlenecks,  but  all  four  cognitive 
bottlenecks  that  were  the  focus  of  the  program.  Table  1  provides  a  summary  of  the  empirical  results 
for  each  industry  team. 
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Table  1 .  IWIIUS  Phase  II  bottleneck  results  for  all  development  teams. 
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Principle  Bottlenecks  assigned  to  a  development  team  are  denoted  by:  Bold  Text 


EXECUTIVE  SUMMARY 


Over  the  last  decade  many  technological  advances  have  been  made  that  affect  how  rapidly 
machines  collect,  store,  and  share  information.  Due  to  this  advancement,  the  information-processing 
capacity  of  humans  has  quickly  become  a  limiting  factor  in  human-computer  interactions.  This 
problem  has  spurred  the  development  of  a  new  scientific  discipline  called  Augmented  Cognition 
(AugCog).  Addressing  methods  for  detecting  and  mitigating  limitations  of  human  information 
processing  and  designing  solutions  to  enhance  exchanging  and  using  information  in  man-machine 
systems  are  the  specific  concerns  of  AugCog.  The  Defense  Advanced  Projects  Agency  (DARPA), 
recognizing  the  operational  implications  of  this  problem  for  the  military,  has  sponsored  the  Improv¬ 
ing  Warfighter  Information  Intake  under  Stress  (IWIIUS)  program.  IWIIUS  is  demonstrating  how 
the  effectiveness  of  warfighters  can  be  improved  by  making  the  human-computer  interface  respond 
to  human  performance  capabilities  and  limitations  in  stressful  operational  environments.  In  Phase  I 
of  the  IWIIUS  program,  the  focus  was  on  developing  and  demonstrating  basic  component  technolo¬ 
gies,  specifically  the  development  of  “cognitive  state  gauges.”  This  development  effort  culminated 
with  a  Technology  Integration  Experiment  (TIE).  The  TIE  incorporated  various  sensor  technologies 
and  processing  algorithms  developed  during  Phase  I  of  the  program  into  a  common  context  that  was 
evaluated  simultaneously  under  the  same  test  conditions,  using  the  same  participants.  The  TIE 
successfully  demonstrated  that  the  technology  of  detecting  cognitive  state  changes  had  matured 
sufficiently  to  monitor  workload  in  an  operationally  relevant  command-and-control-type  decision¬ 
making  task.  The  TIE  report,  issued  in  2003,  provides  an  overview  of  the  physiological  sensors 
and  algorithms  used  during  Phase  I  in  creating  cognitive  state  gauges  to  identify,  in  near  real  time, 
an  assessment  of  cognitive  state. 

This  report  (Volumes  I  and  II)  documents  the  successful  completion  of  Phase  II  of  the  IWIIUS 
program.  The  objectives  of  Phase  II,  described  in  detail  in  this  report,  built  upon  the  cognitive  state 
gauges  created  in  the  first  Phase.  Phase  II  extended  the  work  of  Phase  I  by  focusing  on  the  develop¬ 
ment  of  closed-loop  prototype  AugCog  systems  that  addressed  theoretically  derived  cognitive  bottle¬ 
necks  that  are  a  limiting  factor  in  human-computer  interaction.  Each  industry  team  developed 
the  CLIPs,  which  rely  on  computational  systems  to  determine  the  state  of  the  cognitive  bottleneck 
in  a  warfighter  in  real  time.  Whenever  the  warfighter  approached  detected  cognitive  limits  (minimum 
or  maximum)  of  one  or  more  of  the  cognitive  bottlenecks,  mitigation  strategies  were  implemented, 
given  the  warfighter’s  current  task  requirements,  to  reallocate  resources  or  redirect  efforts  of  the 
warfighter  to  enhance  overall  performance.  The  mitigations  invoked  controlled  the  information 
provided  to  and  from  the  operator.  Optimizing  the  warfighter’s  cognitive  state  through  these  mitiga¬ 
tion  strategies  reduces  the  cognitive  bottlenecks  to  maximize  information  throughput  with  the 
warfighter,  and  thereby  enhances  overall  human-computer  system  performance.  Four  independent 
industry  teams  used  arrays  of  physiological  and  environmental  sensors  tailored  to  the  operational 
requirements  of  four  distinct  warfighting  applications  and  environments.  Each  team  investigated 
a  different  operationally  relevant  task,  specifically  addressing  one  of  the  cognitive  processing 
bottlenecks  identified  at  the  start  of  Phase  II  by  the  DARPA  management  team.  The  DARPA 
management  team  established  performance  goals  for  each  cognitive  processing  bottleneck  as  criteria 
for  success  in  Phase  II.  Table  2  lists  each  industry  team,  their  military-relevant  task  area,  specific 
bottlenecks  investigated,  and  transition  sponsor.  The  bottlenecks,  metrics,  and  key  technical 
approaches  are  identified  in  Figure  1. 
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Table  2.  Phase  II  cognitive  information-processing  challenges. 


Industry  Team 

Military  Application 

Transition  Sponsor 

Primary  Bottleneck 

Honeywell®1 

Dismounted  soldier 

Department  of  the  Army 

Attention 

Daimler 

Chrysler" 

Armored  vehicle  driver 

U.S.  Marine  Corps 

Sensory  Input 

Lockheed 
Martin®  ATL 

Tactical  Strike 
Coordinator  (TSC) 

ONR 

Working  Memory 

Boeing"' 

UCAV  operator 

U.S.  Air  Force 

Executive  Function 

The  ultimate  goal  of  this  program  is  to  demonstrate  significant  operational  benefits  and  consequent 
military  impacts,  including  (1)  enhanced  operational  effectiveness  through  specific  improvements 
to  warfighter  efficiency,  (2)  an  increase  in  the  amount  of  information  that  operators  can  handle, 

(3)  demonstrated  ability  to  reduce  manpower  requirements  (e.g.,  one  person  doing  the  job  of  two  or 
more),  and  (4)  improving  attention  management  during  stressful  operations. 


Challenge:  100% 

Attention  Bottleneck 
Key  Technical  Idea 

Attention  Management 


Enhance  attention 
management  via  a 
directed  attention  and 
autonomous  task 
delegation  strategy 


Challenge:  100% 

Executive  Function  Bottleneck 


Key  Technical  Idea 

Cued  Memory  Retrieval 


Maximize  executive 
functioning  and  facilitate 
memory  enhancement  via 
an  automatic  cued 
retrieval  strategy 


Key  Technical  Idea 

Multimodal  Systems 


•  Exploit  multiple  sensory  channels  via  an 
autonomous  information  delivery  strategy  to  multiple 
modalities 


Challenge:  500% 

Working  Memory 
Key  Technical  Idea 

Sequential  Processing 

•  Maximize  working  memory  processes 
via  an  autonomous  intelligent 
interruption  and  negotiation  strategy 


Figure  1.  Phase  II  cognitive  bottlenecks  and  improvement  goals. 
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Each  industry  team  provided  a  detailed  summary  (see  Section  3)  of  their  accomplishments  in 
defining  and  mitigating  one  or  more  of  the  cognitive  bottlenecks  identified  for  their  chosen 
applications.  A  brief  summary  of  each  team  is  described  below. 

U.S.  Army — Future  Force  Warrior  (FFW) 

The  U.S.  Army  has  defined  the  roles  of  the  2010-era  Future  Force  Warrior  (FFW).  The  FFW 
program  seeks  to  push  information  exchange  requirements  to  the  lowest  levels  and  posits  that 
with  such  enhanced  capabilities  a  squad  would  cover  a  battlefield  in  the  same  way  that  a  platoon 
now  does.  Such  tasking  requires  a  full  range  of  netted  communications  (networked  voice  and  video 
communications  as  well  as  sensor  information)  and  collaborative  situational  awareness  tools 
to  afford  the  FFW  the  required  information  that  would  afford  knowledge  of,  and  coordination 
between,  individual  warfighters.  Providing  improved  information  processing  to  the  FFW  program 
will  enhance  the  overall  capability  of  small  units  of  forces  to  complete  the  complex  mission  task. 
These  enhancements  would  reduce  the  current  number  of  warfighters  required  to  operate  in  more 
complex,  and  less  flexible  operational  units. 

Honeywell"  Laboratories,  selected  as  the  Lead  for  this  effort,  conducted  task  analyses  for  current 
dismounted  soldier  operations  and  identified  several  factors  that  negatively  impacted  battlefield 
performance  of  ground  infantry  units.  One  prime  factor  inhibiting  current  operations  was  communi¬ 
cation  efficiency,  a  limitation  in  the  ability  of  individual  warfighters  to  attend  to  the  most  crucial 
communications  during  critical  mission  events.  Problems  in  attention  have  dire  consequences  in  the 
infantry  operational  environment.  Implications  of  communication  inefficiency  include  the  loss  of 
mission-critical  information  communicated  within  and  between  units.  The  Honeywell"  team 
examined  various  methods  and  procedures  to  address  these  problems  relating  to  the  Attention 
bottleneck.  This  effort  specifically  addresses  one  concern  of  the  U.S.  Army  related  to  the  anticipated 
increase  in  information-processing  requirements  resulting  from  deployment  of  netted 
communications. 

The  Honeywell®  AugCog  team  consisted  of  the  collaborative  efforts  of  Honeywell  "  Laboratories, 
Carnegie  Mellon  University,  City  College  of  New  York,  Clemson  University,  Columbia  University, 
Human  Bionics,  Institute  of  Human  and  Machine  Cognition,  Oregon  Health  and  Sciences  University, 
and  UFI.  This  team  developed  the  Joint  Human-Automation  AugCog  System  (JHAAS)  for  applica¬ 
tion  to  the  U.S.  Army’s  FFW  program.  This  system  exploited  real-time  neurophysiological  and 
physiological  measurements  of  the  human  operator  to  augment  the  battlefield  environment 
to  improve  human-computer  joint  performance.  The  JHAAS  identified  specific  real-time  cognitive 
state  gauges  of  the  human  operator’s  performance  for  measuring  attentional  augmentation. 

The  Honeywell"  team  adopted  an  approach  that  considers  the  joint  human-computer  system  when 
identifying  bottlenecks  to  improve  system  performance.  The  allocation  of  attention  is  critical  to  FFW 
because  it  directly  affects  two  cornerstone  technology  thrusts  of  the  FFW  program:  netted  communi¬ 
cations  and  collaborative  situation  awareness.  In  other  words,  the  appropriate  allocation  of  attentional 
resources  is  critical  for  optimal  operator  performance.  They  operationally  defined  the  Attention 
bottleneck  in  terms  of  two  aspects  of  attention  defined  in  prevailing  cognitive  literature:  the  cognitive 
resources  that  are  available  within  the  warfighter  at  any  given  time  for  assimilating  information,  and 
how  resources  are  allocated  between  competing  information  and  processing  task  demands.  Cognitive 
gauges  were  developed  to  monitor  information  processing  that  was  reflective  of  two  attentional 
bottlenecks.  The  gauges  developed  included  electrocardiogram  (ECG),  electroencephalogram  (ECG), 
galvanic  skin  response  (GSR),  heart  interbeat  interval,  and  pupil  dilation,  measures  that  were 
validated  in  Phase  I.  These  measures  were  integrated  through  a  linear  regression  of  weighted  means 
to  identify  the  operators’  cognitive  state.  The  capability  to  assess  cognitive  state  and  determine  the 
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allocation  of  attention  provided  the  opportunity  to  evaluate  and  adapt,  if  necessary,  the  soldier’s 
current  “interaction”  with  their  task  environment.  These  real-time  cognitive  assessments  could  then 
be  used  to  establish  threshold  levels  to  trigger  adaptive  strategies  for  mitigating  attentional 
bottlenecks. 

Honeywell  "  investigated  how  AugCog  technologies  could  be  an  element  of  the  FFW  ensemble 
and  be  effective  with  infantry  soldiers.  They  conducted  two  Concept  Validation  Experiments  (CVEs) 
with  different  sets  of  experimental  questions.  One  CVE  was  conducted  in  a  simulation  environment 
and  focused  on  complete  sensor  integration,  development  of  multiple  mitigation  strategies  to  enhance 
performance,  and  hardware  integration  for  testing  before  moving  to  the  field.  A  second  CVE 
was  conducted  in  a  motion-capture  virtual  environment  laboratory  to  enable  data  from  physiological 
and  neurophysiological  sensors  to  be  collected  on  a  physically  active  individual  who  moved  around 
the  laboratory  in  a  manner  more  consistent  with  how  a  soldier  would  move  in  the  field.  Both  environ¬ 
ments  included  tasks  that  were  relevant  to  the  dismounted  soldier,  including  navigating  to  an 
objective,  identifying  friend  and  foe,  eliminating  foes,  and  communicating  messages  at  the  squad 
and  platoon  level.  Mitigation  strategies  included  communications  scheduling,  tactile  navigation 
cueing  system,  task  offloading  with  a  MEDEVAC  (medical  evacuation)  briefing  tool,  and  an  auto¬ 
mated  target  identification  assistance  tool.  Task  performance  of  the  individual  warfighter  was  the 
primary  dependent  variable  when  comparing  the  AugCog  condition  against  performance  in  an 
unmitigated  baseline  condition.  The  specific  dependent  measures  used  to  assess  performance 
depended  on  where  the  CVE  was  conducted,  and  mitigation  was  used.  The  central  hypothesis 
of  both  CVEs  conducted  was  that  using  mitigations  would  improve  performance  without 
decrementing  performance  of  other  concurrent  tasks. 

Four  mitigation  strategies  were  evaluated  by  the  Honeywell  ®  team  to  address  the  Attention 
bottlenecks.  The  experimental  results  showed  that  performance  can  be  improved  by  380% 
for  the  Attention  Resources  bottleneck  when  mitigated  with  a  navigation  cueing  system,  and  100% 
for  the  Attention  Allocation  bottleneck  when  mitigated  through  a  communications  scheduler. 

The  navigational  cue  system  used  an  innovative  tactile  belt  that  signaled  the  direction  they  needed 
to  go  to  reach  objective  points.  The  communications  scheduling  mitigation  involved  delaying 
the  presentation  of  critical  communications  when  the  system  detected  high  levels  of  activity  in  those 
areas  of  the  brain  associated  with  verbal  processing.  Other  mitigations  were  also  employed  to  address 
the  Attention  and  Working  Memory  bottlenecks,  including:  offloading  procedures  and  tasks  from 
the  warfighter  to  automation  systems,  utilizing  mixed-initiative  automation  assistance  to  recode 
critical  communications  to  alternative  displays  (a  text  display  on  a  handheld  computer  for  instance 
vice  audio-verbal  displays  on  audio  communication  circuits),  and  supplementing  displays  with 
additional  multi-modal  cues  for  time  critical  communications  to  highlight  objects  relating  to  the 
critical  communications.  For  example,  during  high-workload  periods,  the  Communications  Scheduler 
changed  the  manner  in  which  verbal  communication  messages  were  delivered.  The  Communications 
Scheduler  escalated  and  highlighted  high-priority  messages,  and  deferred  low-priority  messages  by 
sending  them  to  the  tablet  PC.  By  doing  this  at  the  appropriate  time  as  determined  by  sensor  gauge 
outputs,  operational  metrics  improved  significantly,  100%  improvement  in  message  comprehension 
and  125%  improvement  in  overall  situation  awareness.  In  addition,  85%  of  participants  reported  that 
communication  was  significantly  easier  when  completing  tasks  when  the  mitigations  were  turned  on. 
The  tactile  navigation  cueing  device  and  the  MEDEVAC  negotiation  tool  also  significantly  enhanced 
participant  performance,  as  evidenced  by  380%  improvement  in  avoiding  enemy  encounters, 
no  negative  effect  on  ability  to  identify  and  eliminate  foes,  and  no  negative  effect  on  subjective 
workload.  Eighty-percent  of  participants  reported  navigation  tasks  were  easier  when  mitigation  was 
used.  In  addition,  a  96%  improvement  in  communication  of  critical  information,  303%  improvement 
in  time  to  complete  negotiation,  and  no  negative  effect  on  ability  to  identify  or  eliminate  foes, 
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with  no  negative  effect  on  subjective  workload.  While  these  performance  gains  are  impressive, 
the  continuous  use  of  these  strategies  would  come  at  the  cost  of  a  loss  of  situation  awareness  and 
survey  knowledge  of  the  environment.  Such  a  result  highlights  the  significant  contribution  of  the 
IWIIUS  program  by  demonstrating  the  importance  of  modifying  task  demands  based  on  a  user’s 
cognitive  state  to  enhance  performance  and  that  these  mitigations  must  only  be  triggered  when 
the  immediate  benefits  outweigh  the  long-term  costs  to  building  good  situation  awareness  and  survey 
knowledge  of  the  environment. 

The  Honeywell R  team  exceeded  the  objectives  of  Phase  II  by  demonstrating  that  a  closed-loop 
computational  system  can  improve  performance  in  a  simulated  dismounted-military  environment. 
They  showed  that  when  appropriate  mitigations  were  applied  at  appropriate  times,  significant 
improvement  in  attentional  performance  was  obtained. 

Based  on  post-hoc  analyses  and  comparison  with  critical  events  embedded  in  the  test  scenarios 
where  high  attentional  demands  were  expected,  the  attentional  bottlenecks  gauges  appeared 
to  accurately  reflect  high  attention  demands  98%  of  the  time.  The  Honeywell  "  cognitive  gauges 
detected  state  changes  within  250  to  300  ms  of  the  onset  of  the  critical  events,  and  mitigations 
were  invoked  within  1  second  of  the  critical  event.  These  improvements  are  well  within  the  IWIIUS 
system  performance  goals  established  for  Phase  II.  Table  3  provides  a  summary  of  the  IWIIUS  Phase 
II  experimental  results  for  the  Honeywell*  team. 
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Table  3.  Phase  II  bottleneck  results  for  the  Honeywell®  team. 


Bottleneck 

Gauge 

Operational 

Definition 

Phenomenological 

Basis 

Sensor 

Sources 

Classification 

Method 

Mitigation 

Strategy 

Bottleneck 

Metric 

Result 

Classification 

Accuracy 

(>90%) 

State 

Detection 

Time 

(<2s) 

Mitigation 
ON  (< 
lmin) 

Attention  - 
Resources 

Fluctuations 
in  capacity 
of 

attentional 

resources 

Autonomic 
nervous  system 
activity,  Arousal, 
and  Stress 

IBI, 

ECG, 

Pupil, 

GSR 

Spectral 

analysis 

techniques; 

linear 

regression  on 

weighted 

means 

Tactile 

Navigation 

Cueing 

(directional 

cueing) 

380% 

98% 

<300ms 

<ls 

Attention  - 
Allocation 

Fluctuations 
in  allocation 
of  attentional 

resources 

Relative  EEG 
bands  between  two 
frontal  midline  sites 
(FPz,  CPz), 
Engagement  index 
=  beta  /  alpha  + 
theta),  &  P300 

EEG 

Spectral 

analysis 

techniques 

Communications 

Scheduler 

100% 

98% 

<250ms 

<ls 

Working 

Memory 

Working 
memory 
capacity 
within  multi¬ 
tasking 
environment 

Autonomic  nervous 
system  activity, 
Arousal  (PNS  from 
IBI); 

IBI, 

ECG, 

EEG 

Spectral 

analysis 

techniques 

Communications 

Scheduler 

155% 

90% 

250-300 

ms 

<ls 
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U.S.  Marines — Light  Armored  Vehicle  (LAV) 

DaimlerChrysler  Corporation  (DCC)  was  selected  to  participate  as  an  industry  team  leader 
for  the  Marine  Light  Armored  Vehicle  (LAV)  application.  Their  status  in  the  IWIIUS  program 
is  unique  in  that  they  came  into  the  program  with  a  well-defined  application  and  many  previously 
defined  mitigation  possibilities  for  augmentation.  Their  participation  in  IWIIUS  allowed  them 
to  explore  incorporating  cognitive  state  measures  into  their  previously  defined  driver  augmentation 
systems,  as  well  as  consider  augmentation  directly  targeted  at  improving  human  cognition  during 
driving  maneuvers.  DCC  has  a  long  history  as  leading  developers  of  vehicle  and  traffic  safety 
systems,  as  well  as  innovative  vehicle  safety  systems,  making  them  uniquely  qualified  to  address 
how  augmented  cognition  might  be  useful  to  future  fighting  vehicles. 

The  DCC  team  consisted  of  researchers  and  engineers  from  DaimlerChrysler®,  University 
of  Pittsburgh,  Fraunhofer  Institute  FIRST,  Sandia  National  Laboratories,  and  Transferzentrum 
fur  Mikroelektronik.  The  team  concentrated  on  the  integration  of  an  operator  sensor  suite  composed 
of  EEG,  a  novel  seat  posture  detection  system  developed  in  Phase  I,  ECG,  audio  monitoring  of 
vehicle  occupants,  and  driver  behavior  as  monitored  through  vehicle  control  sensors.  DCC  collected 
extensive  situational  data  from  the  vehicle  to  assess  the  driving  context  in  which  the  secondary 
cognitive  tasks  were  performed  as  well  as  data  about  the  driver’s  momentary  condition  in  terms 
of  sensory  and  cognitive  processing. 

Data  acquisition  included  EEG,  participants’  verbal  responses,  reaction  times,  and  recording 
vehicle-related  data  via  an  extended  Controller  Area  Network  (CAN-bus)  system.  Cognitive 
workload  was  assessed  in  real  time  by  two  classifiers:  (1)  two  independent  EEG-based  classifiers 
trained  (neural-network)  to  detect  task-specific  neural  signal  patterns,  and  (2)  a  context-based 
classifier  trained  to  detect  potentially  demanding  driving  and  traffic  situations.  For  quantifying 
the  AugCog  system’s  success,  participant  reaction  times  and  response  accuracy  on  the  competing 
tasks  were  also  evaluated  and  combined  to  yield  an  overall  operator’s  performance  score.  The  same 
data  were  used  to  dynamically  assess  driver  stress  levels. 

Raw  gauge  and  context  data  were  processed  into  various  state  gauges  through  dedicated  processing 
algorithms,  and  the  results  broadcast  across  the  vehicle’s  CAN-bus  architecture.  The  physiological 
sensor  data  were  processed  using  a  linear  discriminant  analysis  to  assess  the  cognitive  state,  while 
vehicle  sensor  data  were  processed  through  a  neural  network  to  generate  context  gauges.  The  various 
gauge  inputs  were  then  processed  through  a  rule-based  system  to  determine  whether  augmentation 
was  required,  and  what  mitigation  was  appropriate  to  address  the  two  sensory  bottlenecks  and  the 
Working  Memory  bottleneck. 

Two  mitigation  strategies  were  investigated:  modality  switching,  and  queuing.  Modality  switching 
involved  presenting  a  discrete  information-processing  task  using  the  visual  sensory  modality  when 
the  cognitive  gauges  indicated  the  auditory  sensory  bottleneck  was  saturated  and  there  was  available 
visual  sensory  capacity,  or  using  the  auditory  channel  when  the  visual  channel  was  at  capacity.  When 
both  bottlenecks  were  at  capacity,  queuing  was  invoked  to  delay  the  presentation  of  tasks  until  one 
of  the  bottlenecks  was  available  to  process  information. 

The  goal  of  the  CVE  was  to  show  that  despite  the  continuous  high  demands  on  visual  sensory 
input  imposed  by  driving  a  vehicle,  the  operator  could  process  significantly  more  information  when 
it  was  presented  as  sensory  capacity  was  made  available.  To  tax  the  sensory  processing  associated 
with  auditory  sensory  input,  a  second  primary  navigation  task  was  created  in  which  situational  briefs 
were  provided  on  a  continuous  basis,  with  periodic  queries  regarding  previously  presented  informa¬ 
tion.  Competing  secondary  tasks  required  the  monitoring  of  auditory  driving  directions  and  mental 
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arithmetic  tasks  that  created  secondary  demands  on  auditory  and/or  visual  processing  as  well  as 
working  memory. 

The  experimental  protocol  was  carried  out  twice  for  each  participant  (n  =  5) — once  with  and  once 
without  the  mitigation  strategy  employed.  Tasks  were  designed  to  specifically  test  the  Auditory 
Sensory  Input  and  Visual  Sensory  Input  bottlenecks  with  the  goal  of  improving  throughput  of 
sensory  input  at  least  100%.  In  addition,  DCC  elected  to  address  the  Working  Memory  bottleneck 
to  demonstrate  an  improvement  in  performance  in  this  environment. 

Results  indicated  that  overall  operator  performance  improved  significantly  on  the  competing  tasks 
when  the  closed-loop  augmentation  system  was  activated.  Without  the  intervention,  competing  task 
performance  was  significantly  lower  during  periods  of  high  workload.  The  empirical  results  demon¬ 
strated  improvements  of  108%  for  the  Sensory-Auditory  bottleneck,  meeting  the  IWIIUS  program 
goal  for  the  Sensory  bottleneck.  DCC  also  achieved  a  72%  improvement  for  the  Visual-Sensory 
bottleneck;  this  improvement  is  still  highly  significant,  given  that  they  were  using  real-world  driving 
tasks  on  public  roads,  which  are  highly  demanding  for  the  Visual-Sensory  bottleneck,  and  therefore 
leave  little  available  capacity  for  competing  visual  tasks.  The  Working  Memory  bottleneck  showed 
a  103%  improvement  over  baseline  (non-mitigated)  conditions.  Furthermore,  driving  performance 
was  assessed  as  the  primary  task  through  a  number  of  safety-relevant  parameters  like  distance 
to  vehicle  ahead,  accelerating  and  braking  events,  and  steering  movements,  etc. 

The  comparison  of  results  for  detecting  high-demand  periods  for  the  sensory  bottlenecks  and 
Working  Memory  from  the  cognitive  state  gauges  was  compared  to  the  context  data  derived  from 
vehicle  dynamics  sensors  to  allow  the  accuracy  and  speed  of  cognitive  state  detection  to  be  analyzed. 
The  visual  sensory  bottleneck  was  accurately  detected  more  than  95%  of  the  time  within  0.5  second, 
and  the  auditory  sensory  memory  was  detected  up  to  95%  of  the  time  within  0.2  second.  High 
periods  of  working  memory  load  were  detected  correctly  up  to  95%  of  the  time  within  0.2  second. 
Mitigations  were  triggered  well  under  the  IWIIUS  program  goal  of  less  than  1  minute,  in  as  little 
as  0.12  second  to  no  more  than  10  seconds,  depending  on  the  mitigation  and  trigger  event  that 
was  called  for  in  the  CLIP  logic. 

The  success  of  the  DCC  demonstration  is  expected  to  serve  as  the  basis  for  improvement 
in  performing  competing  mission  tasks  and  dynamic  task  allocations  of  military  vehicle  operators 
in  future  demonstrations.  The  DCC  team  successfully  demonstrated  that  it  is  possible  to  detect 
cognitive  state  changes  and  that  this  information  provides  significant  improvement  in  operator 
performance  under  closed-loop  conditions.  Table  4  provides  a  summary  of  the  IWIIUS  Phase  II 
experimental  results  for  the  DCC  team. 
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Table  4.  Phase  II  bottleneck  results  for  the  DCC  team  1 . 


Bottleneck 

Gauge 

Operational 

Definition 

Phenomenological  Basis 

Sensor 

Source(s) 

Classification 

Method 

Mitigation 

Strategy 

Bottleneck 

Metric 

Result 

Classification 

Accuracy 

(>90%) 

State 
Detection 
Time  (< 

2s) 

Mitigation 
ON  (<1 
min) 

Sensory 

Input 

(Auditory) 

Auditory 

Engagement 

Decrease  in  Power  in 

Alpha  Band  -  localization 
is  task  specific  -  LDA 
classifier  does 
localization 

EEG  + 

Context 

(Vehicle 

Sensors 

and  Seat 

Posture) 

LDA  -  Linear 
Discriminate 
Analysis  for 

EEG  and 

Neural 
network  for 
Context  -  sent 
to  mitigation 
logic 

1)  Modality 
switch,  2) 
Scheduling 
given  task 
requirements 

108% 

>95% 

200ms 

<ls 

Sensory 

Input 

(Visual) 

Visual 

Engagement 

Decrease  in  Power  in 

Alpha  Band  -  localization 
is  task  specific  -  LDA 
classifier  does  localization 

EEG  + 

Context 

(Vehicle 

Sensors 

and  Seat 

Posture) 

LDA  -  Linear 
Discriminate 
Analysis  for 

EEG  and  Neural 
network  for 
Context  -  sent  to 
mitigation  logic 

1)  Modality 
switch,  2) 
Scheduling 
given  task 
requirements 

72% 

>95% 

200ms 

<ls 

Working 

Memory 

Verbal/Semantic 
Processing  - 
Mental  Arithmetic 

Decrease  in  Power  in 

Alpha  Band  -  localization 
is  task  specific  -  LDA 
classifier  does  localization 

EEG 

LDA  -  Linear 
Discriminate 
Analysis  for 

EEG 

Scheduling 

103% 

>95% 

200ms 

<ls 

The  long-range  implications  of  using  DCC  augmented  cognition  technology  in  the  LAV 
environment  will  be  twofold:  the  workload  of  the  crew  could  be  dynamically  allocated  between 
the  driver,  vehicle  commander,  and  additional  vehicle  occupants  (1)  by  better  sharing  of  various 
tasks  based  on  the  cognitive  state  of  each  crew  member  and  the  autonomous  capabilities  of  the 
vehicle,  (2)  by  using  the  augmented  cognition  system  to  focus  the  crew’s  attention  on  the  most 
critical  tasks,  and  (3)  by  having  the  augmented  cognition  system  report  on  the  cognitive  status 
of  crewmembers  (during  over-  and  under-workload  conditions).  Furthermore,  the  driver’s  ability 
to  safely  operate  the  vehicle  would  be  increased  by  (1)  automating  and  assisting  in  driver  functions 
when  appropriate,  and  (2)  prioritizing  caution  and  warning  indicators  through  the  most  effective 
sensory  channel.  These  applications  will  be  explored  as  part  of  Phase  III. 

U.S.  Navy — Tactical  Tomahawk  Weapons  Control  System  (TTWCS)  for  the  Tactical  Strike 
Coordinator  (TSC) 

Lockheed  Martin®  Advanced  Technology  Laboratory  (LMATL)  was  selected  as  the  industry  team 
lead  for  the  Command  and  Control  environment.  They  conducted  a  CVE  using  a  Tactical  Tomahawk 
Weapons  Control  System  (TTWCS)  simulation  environment.  The  task  environment  that  was  used 
for  the  CVE  was  targeted  at  the  Tactical  Strike  Coordinator  (TSC),  a  single  operator  who  performs 
one  of  the  most  cognitively  demanding  jobs  in  the  Navy,  and  a  position  that  is  expected  to  become 
more  demanding  as  the  TTWCS  mission  evolves1.  The  TSC  must  manage  missile  strikes  from 
multiple  platforms  against  numerous  targets  in  the  operational  theater.  Extremely  high  cognitive 
demands  are  placed  on  the  operator’s  working  memory.  The  operator  must  “flip”  through  multiple 
display  views  to  collect  information  needed  to  make  retargeting  decisions.  This  application  is  a  clear 
choice  for  assessing  the  ability  to  mitigate  the  Working  Memory  bottleneck  that  occurs  during 

1  This  job  is  so  difficult  that  it  is  the  one  operational  billet  in  the  Navy  where  a  new  operator  must  perform  a  full 
deployment  with  his/her  predecessor  to  gain  adequate  proficiency  before  being  allowed  to  do  the  job  alone! 
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the  operator’s  task  performance.  The  LMATL  team  conducted  a  cognitive  task  analysis  of  the  TSC 
position  and  concluded  that  the  demands  on  working  memory  of  the  TSC  are  highly  consistent  with 
the  Working  Memory  bottleneck  outlined  by  the  DARPA  management  team.  The  team  was  lead  by 
LMATL  collaborating  with  Design  Interactive,  NovaSol,  Anthrotronix,  Advanced  Brain  Monitoring, 
Drexel  University,  University  of  Virginia,  and  Eyetracking,  Inc. 

The  LMATL  team  tested  a  variety  of  sensor  technologies  as  inputs  to  developing  working  memory 
gauges.  Several  of  the  gauge  technologies  supported  during  Phase  I  of  this  program  were  used 
at  different  stages  of  development.  The  final  sensor  suite  included  EEG  from  nine  wireless  EEG 
sensors  as  well  as  pupillometry  data  on  pupil  size  for  each  eye,  which  served  as  the  basis  for  a  stand¬ 
alone  Index  of  Cognitive  Activity  (ICA)  gauge.  EKG  and  GSR  were  also  collected. 

Based  on  a  cognitive  task  analysis  of  the  TSC  position  and  a  widely  recognized  theory  of  working 
memory  known  as  multiple-resource  theory  (Wickens,  1984),  the  LMATL  team  operationally 
defined  the  Working  Memory  bottleneck  as  consisting  of  two  principle  channels:  visual-spatial 
information  and  auditory-verbal  information.  Their  approach  was  to  detect  when  an  operator’s 
cognitive  demands  were  high  using  the  ICA,  and  then  to  use  EEG  to  localize  what  channel  was  most 
likely  contributing  the  high  working  memory  demands.  Based  on  this  information,  an  Intelligent 
Sequencing  mitigation  strategy  was  turned  on.  In  this  strategy,  non-critical  alerts  were  held  in  a 
queue  during  physiologically  determined  higher  workload  situations  and  re-scheduled  to  occur 
during  lower  workload  situations.  Critical  alerts  would  be  presented  through  that  channel  that 
was  least  heavily  loaded. 

The  LMATL  architecture  evolved  into  a  system  that  should  prove  robust  for  future  development. 
The  final  gauge  suite  used  only  the  raw  data  feeds  from  each  of  the  individual  sensors  rather  than 
independent  processors  providing  derivative  “gauge”  values,  as  was  seen  in  IWIIUS  Phase  I.  All 
gauge  values  were  computed  through  a  Cognitive  State  Assessor  (CSA)  that  served  as  an  integrated 
working  memory  gauge  processor.  The  CSA  employed  a  neural  network  to  generate  gauge  values 
for  verbal  and  spatial  working  memory.  These  working  memory  gauge  values  were  then  sent  to  the 
PACE  (Performance  Augmentation  through  Cognitive  Enhancement)  processor  to  manage  tasks 
and  enable  the  mitigation  strategy,  which  in  the  current  implementation,  consisted  of  intelligent 
sequencing. 

The  LMATL  team  ran  15  volunteers  in  a  simulated  TTWCS  task  in  two  separate  conditions — once 
with  the  mitigation  turned  on  and  once  while  it  was  off.  Results  indicate  a  642%  improvement  for  the 
Working  Memory  bottleneck  as  measured  by  successful  retargeting  performance  when  the  intelligent 
sequencing  mitigation  was  employed.  The  neural  network  successfully  classified  working  memory 
as  high  or  low  over  90%  of  the  time  based  on  the  workload  periods  designed  into  the  simulation. 

The  workload  classifications  took  place  in  under  0.5  second,  and  the  mitigation  was  triggered 
5  seconds  from  the  time  a  working  memory  exceeded  threshold.  The  LMATL  development  CLIP 
met  all  DARPA  requirements  for  successfully  addressing  the  Working  Memory  bottleneck.  Table  5 
provides  a  summary  of  the  Phase  II  quantitative  results  for  the  Lockheed  Martin®  team. 

These  results  (see  Table  5)  are  operationally  relevant  to  future  TTWCS  development  since 
the  added  dimension  of  retargetable  Tomahawk  missiles,  in  addition  to  the  pressure  for  manning 
reduction,  will  dramatically  increase  the  frequency  with  which  TTWCS  operators  will  experience 
working  memory  overload.  Efforts  are  currently  underway  to  adapt  sensors  that  require  direct  contact 
with  the  skin  to  be  fitted  within  a  helmet/hat  that  could  be  worn  by  TTWCS  operators  for  an  extend¬ 
ed  period  in  an  operational  environment. 
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Table  5.  Phase  II  bottleneck  results  for  the  Lockheed  Martin®  ATL  team. 


Bottleneck 

Gauge 

Operational 

Definition 

Phenomenological 

Basis 

Sensor 

Sources 

Classification 

Method 

Mitigation 

Strategy 

Bottleneck 

Metric 

Result 

Classification 

Accuracy 

(>90%) 

State 

Detection 

Time 

(<2sec) 

Mitigation 

ON 

(<lmin) 

Working 

Memory 

verbal  and 

spatial 

processing 

Fuse  sensors:  ICA 
and  other 
physiological 
sensors  (GSR,  EKG) 
provides  general 
work  load,  EEG 
provided 
localization  of 
verbal/spatial 
working  memory. 

EEG,  GSR, 

Pupillometry, 

EKG 

Neural  Net 

Intelligent 
Sequencing 
(through 
rule  based 
algorithm) 

642% 

90.10% 

.5s 

5s 

U.  S.  Air  Force — Unmanned  Air  Vehicle  Control  Workstation 

Unmanned  Air  Vehicles  (UAVs)  have  become  highly  valued  assets  in  modern  military  operations. 
Current  UAVs  are  operated  with  several  operators  working  as  supervisor  controllers  for  a  single 
vehicle.  Current  operational  goals  for  the  next  generation  of  unmanned  combat  air  vehicles  call 
for  the  ratio  of  operator  to  vehicles  to  reverse,  that  is,  having  multiple  vehicles  that  can  be  operated 
by  a  single  individual.  A  single  operator  managing  multiple  vehicles  must  process  a  very  high 
number  of  information  elements  on  an  ongoing  basis,  which  is  why  the  development  team  focused 
on  Executive  Function  as  their  primary  bottleneck. 

Due  to  their  extensive  background  in  the  operation  and  development  of  the  UAV  control  station, 
Boeing  ®  was  selected  as  the  industry  leader  for  a  team  of  researchers  exploring  the  Executive 
Function  bottleneck  in  the  context  of  the  anticipated  requirements  of  next-generation  UAVs  such 
as  the  Joint  Unmanned  Combat  Air  System  (JUCAS)  and  its  associated  Mission  Control  Station 
(MCS).  The  Boeing®  team  conducted  cognitive  task  analyses  of  several  combat  UAV  missions  and 
determined  that  a  number  of  cognitive  bottlenecks  could  be  addressed  in  the  context  of  a  Suppression 
of  Enemy  Air  Defenses  (SEAD)  mission  as  the  application  environment.  Their  goal  was  to  develop 
a  combat  UAV  system  in  which  a  single  vehicle  operator  would  effectively  manage  multiple  combat 
UAVs  simultaneously. 

This  tasking  required  that  the  Boeing®  team  learn  how  critical  decision  information  is  presented 
to  an  operator,  allowing  them  to  quickly  assimilate  new  information,  integrate  this  information  with 
previously  learned  information  for  a  particular  UAV,  and  then  determine  which  UAVs  had  priority 
for  succeeding  task  management.  While  all  four  of  the  conceptual  bottlenecks  identified  for  study 
during  IWIIUS  Phase  II  are  critical  to  the  combat  UAV  controller,  the  predominant  factor  was  the 
strategic  processing  associated  with  the  Executive  Function  bottleneck.  Although  this  was  their 
specific  objective  for  Phase  II,  they  attempted  to  address  each  DARPA-identified  bottleneck 
(Sensory,  Attention,  Executive  Function,  and  Working  Memory).  The  UAV  application  environment 
is  well  suited  for  the  manipulation  of  cognitive  taskload.  Controlling  multiple  UAVs  is  cognitively 
demanding  because  it  forces  the  operator  to  frequently  shift  context  within  and  between  tactical  and 
system  health  tasks  for  each  vehicle  and  among  all  the  vehicles.  The  Boeing®  team  collaborated  with 
Quasar,  EyeTracking,  Inc.,  the  Air  Force  Research  Laboratory,  and  NovaSol,  given  their  experience 
with  EEG,  ECG,  electrooculograph  (EOG),  functional  near-infrared  (INIR),  and  pupillometry 
sensors,  and  the  creation  of  derivative  cognitive  gauges. 

The  system  architecture  used  a  publish/subscribe  system  where  sensor  data  were  collected  and 
distributed  to  other  system  components  as  needed.  The  CSA  received  raw  sensor  data  from  the 
publish/subscribe  system.  Cognitive  gauge  values  were  then  computed  and  published  through  the 
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same  system.  An  Augmentation  Manager  received  the  output  of  the  cognitive  gauges  and  combined 
it  with  context  data  that  were  derived  from  operator  behaviors  and  a  computation  of  the  number 
of  impending  critical  tasks  through  a  rule-based  system  within  the  simulation.  The  Augmentation 
Manager  turned  mitigations  on  and  off  based  on  the  joint  function  of  the  cognitive  gauge  output  and 
of  the  mission  context  data. 

Two  different  approaches  were  taken  in  implementing  the  real-time  cognitive  state  assessor. 

The  initial  approach  was  based  on  a  neural  network  that  received  EEG,  ECG,  EOG,  pupillometry, 
and  fNIR  sensor  data.  The  neural  network  was  used  to  detect  Attention  and  Executive  Function 
bottlenecks  and  was  trained  to  recognize  high  and  low  cognitive  taskload  for  the  target  prosecution 
task.  The  augmentation  manager  presented  mitigations  when  the  artificial  neural  network  indicated 
that  target  prosecution  taskload  was  high  and  the  task  context  assessor  indicated  that  a  backlog 
was  developing  or  that  a  pop-up/time-critical  target  had  been  detected.  Mitigations  for  the  Attention 
bottleneck  were  designed  to  provide  multi-modal  cueing  to  direct  the  operator’s  attention  to  the 
highest  priority  task.  Mitigations  for  the  Executive  Function  bottleneck  were  designed  to  help 
the  operator  quickly  reinstate  context  for  the  primary  target  prosecution  task. 

A  follow-on  approach  in  developing  a  cognitive  state  assessor  was  based  on  a  statistical  process 
control  model  that  in  its  current  implementation  received  only  fNIR  and  pupillometry  sensor  data. 
The  expectation  was  that  this  approach  would  make  the  implementation  of  alternative  mitigations 
for  additional  tasks  easier  in  future  systems.  For  IWIIUS  Phase  II,  the  statistical  process  control 
model  was  used  to  detect  sensory  and  Working  Memory  bottlenecks.  The  Augmentation  Manager 
presented  mitigations  when  the  cognitive  gauges  indicated  that  target  prosecution  and/or  vehicle 
health  taskload  was  high,  when  pupillometry  detected  global  cognitive  workload,  and  when  the  task 
context  assessor  indicated  that  a  backlog  of  target  prosecution  or  vehicle  health  tasks  was  developing. 
Mitigations  for  the  Sensory  bottleneck  were  designed  to  support  task  prioritization,  task  interruption 
management,  and  multi-modal  cueing.  Mitigations  for  the  Working  Memory  bottleneck  were 
designed  to  support  task  prioritization,  task  sequencing,  and  task  interruption  management. 

The  Boeing®  team  conducted  a  series  of  CVEs,  refining  the  bottleneck  gauges  and  mitigation 
strategies  iteratively  based  on  empirical  results  throughout  the  development  process.  A  subset 
of  data  obtained  from  the  CVEs  were  used  in  assessing  the  bottleneck  results  reported  here.  The 
Boeing®  development  effort  met  the  performance  goals  for  all  the  cognitive  bottlenecks,  including 
a  241%  improvement  for  their  targeted  bottleneck  of  Executive  Function.  Comparing  cognitive  state 
detections  for  the  Executive  Function  bottleneck  to  data  from  the  pending  tasks  indicated  that 
the  Executive  Function  bottleneck  was  successfully  categorized  92%  of  the  time,  within  1  second 
of  its  changing.  Changing  the  mitigation  status  was  done  within  1  second  of  being  called  for  by  the 
Augmentation  Manager.  These  results  were  well  within  the  program  goals  of  achieving  a  1 00% 
improvement  in  Executive  Function,  with  90%  or  better  accuracy  in  less  than  2  seconds  after  a  state 
change,  and  mitigating  in  less  than  1  minute.  Table  6  provides  a  summary  of  the  Phase  II  results 
for  the  Boeing®  team.  The  performance  differences  observed  between  the  augmented  and  non- 
augmented  conditions  were  statistically  and  operationally  meaningful. 
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Table  6.  Phase  II  bottleneck  results  for  the  Boeing®  team. 


Bottleneck 

Gauge  Operational 
Definition 

Phenomenological 

Basis 

Sensor 

Sources 

Classification 

Method 

Mitigation 

Strategy 

Bottleneck 

Metric 

Result 

Classification 

Accuracy 

(>90%) 

State 
Detection 
Time  (<2 
s) 

Mitigation 

ON 

(<lmin) 

Executive 

Function 

Shifting  context 
interferes  with 
cognitive  processing 
capacity 

Literature 
suggested  that 
physiological 
correlates  existed 
that  indicated 
(measured) 
cognitive  activity 

EEG/ECG, 

pupillometry 

Neural  Net 

Map 

Declutter, 

Multimodal 
Cueing,  and 

Process 

Queuing 

241% 

92% 

1  sec 

1  sec 

Working 

Memory 

-  Verbal/language 
processing 

-  Task  context 

-  Increased  Activity 
Localized  to 

Wemike  area 
indicates  verbal 
processing. 

-  Increased  pupil 
area  indicates  global 
processing 

-  Task  context 

fNIR, 

pupillometry 

Statistical 

Process  Control 

Sequencing 

680% 

100% 

1  sec 

1  sec 

Sensory 

Input 

-  Verbal/language 
processing 

-  Global  processing 

-  Task  context 

-  Increased  Activity 
Localized  to 

Wemike  area 
indicates  verbal 
processing. 

-  Increased  pupil 
area  indicates  global 
processing 

-  Task  context 

fNIR, 

pupillometry 

Statistical 

Process  Control 

Process 

Queuing, 

Sequencing, 

Multimodal 
cueing,  and 

Map 

Declutter 

283% 

100%  * 

1  sec 

1  sec 

Attention 

Part-Task  Gauge  for 
Tactical  Situation 

Display 

Neural  Net  trained  to 
representative  task 

EEG/ECG, 

pupillometry 

Neural  Net 

Multimodal 
cueing  and 

Sequencing 

750% 

94% 

1  sec 

1  sec 

17 


Conclusion 


Phase  II  of  the  IWIIUS  program  was  an  unqualified  success.  All  four  conceptual  bottlenecks 
identified  at  the  start  of  Phase  II  were  successfully  addressed  by  one  or  more  of  the  development 
teams  in  four  different  military  application  environments.  Each  team  demonstrated  the  ability 
to  detect  high  cognitive  load  within  a  specific  bottleneck  by  using  physiological  measures 
of  cognitive  state.  These  measures  were  then  used  as  a  trigger  to  initiate  an  appropriate  mitigation 
strategy  at  the  appropriate  time,  providing  significant  improvements  in  operator  performance. 

The  magnitude  of  the  performance  enhancements  varied,  depending  on  the  task  environment  and 
bottleneck.  Performance  enhancements  well  in  excess  of  the  DARPA  goals  established  at  the  start 
of  Phase  II  were  realized.  The  IWIIUS  program  continues  on  track,  and  teams  are  primed  and  ready 
to  extend  their  efforts  from  the  laboratory  to  the  real  operational  environment  under  stress  during 
Phase  III  of  the  program. 
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1.  INTRODUCTION 


Humans  are  constrained  in  the  amount  of  information  they  can  manage.  The  goal  of  the 
DARPA  Improving  Warfighter  Information  Intake  Under  Stress  (IWIIUS)  program 
is  to  develop  closed-loop  computational  systems  in  which  the  computer  adapts  the  systems 
operations  based  on  the  systems’  ability  to  detect  the  cognitive  state  of  individual  warfighters. 
Ultimately,  these  systems  will  dramatically  improve  the  human-machine  performance 
of  warfighters  engaged  in  demanding  decision-making  tasks.  In  the  course  of  executing  the 
IWIIUS  program,  it  is  anticipated  that  the  scientific  discipline  of  Augmented  Cognition 
(AugCog)  that  will  be  established  will  become  the  basis  for  continued  scientific  and  technology 
developments.  The  IWIIUS  program  will  demonstrate  significant  operational  benefits  and 
consequent  military  impacts,  including  the  following: 

•  Enhance  operational  effectiveness  through  specific  improvements  to  warfighter  efficiency 

•  Increase  the  amount  of  information  that  operators  can  handle 

•  Reduce  manpower  requirements:  one  person  doing  the  job  of  three 

•  Improve  attention  management  during  stressful  operations 

This  report  (Volumes  I  and  II)  describes  work  performed  under  Phase  II  of  the  IWIIUS 
program.  Phase  II  focused  on  integrating  cognitive  state  gauges  into  operational  systems  that 
could  demonstrate  the  viability  of  augmenting  cognition  into  an  operational  context,  which  is 
discussed  in  this  volume.  Volume  II  (Appendices)  of  this  report  describes  the  efforts  of  four 
industry  teams  who  addressed  four  distinct  military  decision-making  applications.  Each  of  their 
efforts  targeted  a  specific  theoretically  derived  cognitive  bottleneck.  Each  team  was  tasked  with 
building  a  self-contained,  real-time,  AugCog  system  that  would  be  tested  as  a  closed-loop 
integrated  prototype  (CLIP).  Each  team’s  CLIP  was  evaluated  through  a  series  of  experimental 
test  protocols  to  demonstrate  compelling  improvements  in  the  information  throughput  with 
regard  to  the  cognitive  bottleneck. 

Program  Overview.  The  IWIIUS  program  is  being  executed  as  a  multi-phase  accelerated 
development  program  to  allow  development  progress  to  be  periodically  assessed  as  the 
program  matures  and  to  better  manage  development  risks. 

Phase  I.  Phase  I  of  the  program  was  completed  in  2003.  It  focused  on  developing  and 
demonstrating  basic  component  technologies,  specifically  the  development  of  “Cognitive  State 
Gauges.”  This  development  effort  culminated  with  a  Technical  Integration  Experiment  (TIE), 
where  all  the  sensor  technologies  and  processing  algorithms  developed  during  Phase  I  were 
brought  together  in  a  common  test  context  and  evaluated  at  the  same  time,  under  the  same  test 
conditions,  with  the  same  participants.  The  results  of  the  TIE  became  the  basis  for  the  Phase  I 
Final  Report  (St.  John,  Kobus,  and  Morrison,  2003)  and  the  awarding  of  Phase  II  funding. 

Phase  I  involved  the  evaluation  of  20  psychophysiological  derived  measures  (cognitive  state 
gauges)  that  were  developed  under  the  AugCog  program.  These  gauges  came  from  1 1  different 
research  groups  and  were  developed  with  a  variety  of  theories  and  scientific  backgrounds.  The 
TIE  brought  these  disparate  approaches  to  assessing  cognitive  state  together  to  be  assessed  with 
a  common  test  protocol  using  a  relatively  complex  cognitive  task  derived  from  the  real-world 
decision-making  requirements  seen  with  tactical  decision-makers.  The  gauges  used  a  wide 
range  of  sensory  technologies  and  were  based  on  very  different,  yet  sometimes  overlapping, 
theoretical  approaches.  The  sensor  technologies  included  fNIR,  continuous  and  event-related 
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EEG/ERP,  eye  tracking  and  pupil  dilation,  mouse  pressure,  body  posture,  heart  rate,  and  GSR. 
These  technologies  measured  an  array  of  cognitive  states  such  as  vigilance,  arousal  level, 
executive  load,  and  perceptual/motor  load.  Gauges  were  compared  across  data  collection 
teams  for  the  evaluation  of  each  gauge  and  its  ability  to  detect  changes  in  cognitive  activity 
as  it  was  manipulated  using  the  Warship  Commander  Task  (WCT)  during  the  experiment. 

Phase  I  Objectives  included  the  following: 

1 .  Examine  the  state-of-the-art  in  theory  and  technology  that  had  been  or  could  be  used  to 
assess  human  cognitive  states. 

2.  Select  the  most  promising  technologies  and  scientists  as  the  basis  for  DARPA  investment 
to  develop  cognitive  state  gauges,  i.e.,  a  suite  of  one  or  more  physiology-based  sensors 
that  could  be  integrated  with  a  processing  algorithm  to  identify  changes  that  theoretically 
correlate  to  changes  in  psychological  state. 

3.  Assess  the  maturity  of  those  gauges  in  detecting  cognitive  state  changes  and  their  poten¬ 
tial  for  application  to  a  real-time  system  that  would  adapt  to  those  changes  to  maximize 
human  performance  in  man-machine  systems  that  would  interest  the  military. 

4.  Based  on  this  empirical  demonstration  that  cognitive  state  gauges  were  feasible  with  the 
state-of-the-art  technologies  that  had  been  developed,  Phase  I  of  the  IWIIUS  program 
was  successful,  and  DARPA  determined  that  Phase  II  should  be  funded. 

Phase  II.  This  report  documents  the  completion  of  Phase  II  of  the  IWIIUS  program. 

The  successful  conclusion  of  Phase  II  was  defined  as  having  built  a  fully  integrated  system 
where  cognitive  state  gauges  could  detect  changes,  trigger  augmentation,  and  measurably 
mitigate  the  overload  of  one  of  four  conceptual  bottlenecks  defined  based  on  current  cognitive 
theory.  Figure  2  shows  the  four  bottlenecks  and  the  rationale  for  mitigating  them. 
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Challenge  #4 

Attention  Bottleneck 
Key  Technical  Idea 

Attention  Management 

*  Enhance  attention 
management  via  a  directed 
attention  and  autonomous 
task  delegation  strategy 


Challenge  #3 

Sensory  Input  Bottleneck 
Key  Technical  Idea 

Multimodal  Systems 


Challenge  #2 
xecutive  Function  Bottleneck 

Key  Technical  Idea 

Cued  Memory  Retrieval 


Maximize  executive 
functioning  and  facilitate 
memory  enhancement  via  an 
automatic  cued  retrieval 
strategy 


Challenge  #1 

Working  Memory  Bottleneck 
Key  Technical  Idea 

Sequential  Processing 


*  Exploit  multiple  sensory  channels  via  an  autonomous 
infonnation  delivery  strategy  to  multiple  modalities 


9  Maximize  working  memory  processes  via 
an  autonomous  intelligent  interruption  and 
negotiation  strategy 


Figure  2.  Phase  II  cognitive  information-processing  challenges. 

Challenge  Number  1,  the  Working  Memory  bottleneck,  refers  to  the  generally  accepted 
theory  that  human  cognition  can  hold  and  actively  process  a  limited  amount  of  information. 
Phase  I  demonstrated  that  gauges  exist  that  could  detect  processing  associated  with  high  levels 
of  activity  within  working  memory.  Therefore,  it  would  be  possible  to  mitigate  this  bottleneck 
by  manipulating  the  order  and/or  times  when  tasks  requiring  significant  amounts  of  working 
memory  were  presented  to  the  operator.  Tactical  command  and  control  tasks  such  as  the 
Tactical  Tomahawk  Strike  Coordinator  require  recalling  and  integrating  a  large  number 
of  information  elements  in  very  short  time  periods.  The  LMATL  team  addressed  the  Working 
Memory  bottleneck  in  the  context  of  the  Tactical  Tomahawk  Systems  Coordinator  (TSC). 

The  Executive  Function  bottleneck,  Challenge  Number  2,  was  derived  from  the  observation 
that  higher  order  executive  control  may  be  involved  in  moving  information  from  working 
memory  to  short-tenn  memory  and  then  onto  frontal  structures  for  coordinated  decision¬ 
making.  It  was  argued  that  this  bottleneck  might  be  effectively  addressed  through  a  strategy 
of  cued  retrieval,  where  by  facilitating  the  chunking  of  related  information,  and  then  enhancing 
the  operator’s  display  appropriately  during  high  levels  of  decision-making,  more  decisions 
could  be  made  in  short  time  periods  with  greater  accuracy.  UAV  operators,  who  must  recall 
a  large  amount  of  highly  diverse  infonnation  for  a  number  of  similar  vehicles,  perform  such 
a  task.  The  Boeing  R  -led  team  addressed  Executive  Function  as  their  primary  bottleneck. 
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The  third  challenge,  the  Sensory  Input  bottleneck,  was  identified  from  prevailing  cognitive 
theory.  This  bottleneck  was  based  on  the  observation  that  people  can  multi-process  a  signifi¬ 
cant  amount  of  information  when  it  is  presented  through  different  sensory  modalities. 
Therefore,  cognitive  state  gauges  could  be  developed  that  detect  when  one  or  more  sensory 
modalities  were  saturated,  and  the  presentation  of  critical  information  then  shifted  to  an  alter¬ 
nate  modality.  Operating  a  motor  vehicle  is  a  task  requiring  significant  amounts  of  visual 
processing,  and  therefore  was  an  ideal  task  for  demonstrating  the  effectiveness  of  mitigating 
the  Sensory  Input  bottleneck.  The  DCC  team  addressed  Sensory  Input  as  their  primary  bottle¬ 
neck,  with  an  eye  toward  application  in  the  next-generation,  USMC  Light  Armored  Vehicle 
and  the  Marine  Expeditionary  Family  of  Fighting  Vehicles  (MEFFVs). 

Finally,  the  Attention  bottleneck  (Challenge  Number  4)  was  identified  based  on  the  recogni¬ 
tion  that  attending  to  the  right  information  in  the  first  place  is  a  critical  part  of  effective 
decision-making.  Developing  interventions  that  aid  a  warfighter  in  attending  to  the  most 
critical  information  at  any  given  time  could  make  a  significant  difference  in  battle  situations. 
Therefore,  the  target  domain  for  which  the  Attention  bottleneck  was  a  primary  focus  was  the 
infantry  soldier.  The  Honeywell  '  -led  team  addressed  Attention  as  their  primary  bottleneck, 
with  an  eye  toward  application  to  the  U.S.  Army’s  FFW  program. 

Phase  II  Objectives  were  defined  to  include  the  following: 

1 .  Identify  application  environments  with  significant  transition  potential  and  a  suitable 
reliance  on  human  operators  with  significant  cognitive  overloads  so  there  might  be 
potential  for  improvement  with  the  introduction  of  an  augmented  cognition  prototype. 

2.  Operationally  define  the  cognitive  bottlenecks  within  the  context  of  the  chosen 
application  environments,  as  well  as  measurable  performance  metrics  on  which  the 
bottlenecks  could  be  empirically  assessed  with  and  without  augmented  cognition 
technologies. 

3.  Define  prospective  mitigation  strategies  that  had  potential  to  improve  human  cognitive 
abilities  based  on  one  or  more  cognitive  state  change  detection  gauges.  The  strategies 
identified  included  the  following: 

a.  Intelligent  interruption  to  improve  limited  working  memory 

b.  Attention  management  to  improve  focus  during  complex  tasks 

c.  Cued  memory  retrieval  to  improve  situational  awareness  and  context  recovery 

d.  Modality  switching  (i.e.,  audio,  visual)  to  increase  information  throughput 

4.  Develop  prototype-augmented  cognition  systems  that  would  integrate  data  from  various 
cognitive  state  gauges  that  could  be  used  to  initiate  mitigation  strategies  to  enhance 
overall  task  performance. 

5 .  Mature  the  augmentation  prototypes  based  on  the  empirical  results  from  a  series  of 
CVEs,  demonstrating  impacts  on  the  identified  cognitive  bottlenecks  and  consequent 
improvements  in  the  overall  human-machine  performance. 

Scientific  Underpinnings.  Over  the  last  decade,  many  technological  advances  have  been 
made  in  how  machines  rapidly  collect,  store,  and  share  information.  However,  the  information¬ 
processing  capacity  of  humans  has  evolved  little  during  this  time  and  has  quickly  become  the 
limiting  factor  in  human-computer  interactions.  The  AugCog  mission  is  to  alleviate  cognitive 
bottlenecks  and  reduce  the  human-computer  interaction  limitations  that  directly  affect  military 
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operations.  Cognitive  bottlenecks  are  potential  interferences  with  an  operator’s  ability 
to  process  and  perform  multiple  tasks  simultaneously.  The  intrinsic  restrictions  in  the  number 
of  mental  tasks  that  a  person  can  execute  at  one  time,  and  this  capacity  itself,  may  fluctuate 
from  moment  to  moment,  depending  on  a  host  of  factors,  including  mental  fatigue,  novelty, 
boredom,  and  stress.  These  limitations  include  attention,  memory,  learning,  comprehension, 
visualization  abilities,  and  decision-making. 

Working  memory  applies  to  a  limited-capacity  system  that  can  store  and  manipulate  infor¬ 
mation  (Baddeley,  1996).  Working  memory  is  significant  in  military  situations  because 
of  the  limited  information  capacity  and  persistence  that  leads  to  a  cognitive  bottleneck,  and 
therefore  limited  human  performance.  For  instance,  overloads  in  competing  tasks  could  lead 
to  a  pilot  overlooking  the  presence  of  a  threat  and  therefore  result  in  an  inappropriate  maneuver 
(Pew  and  Mavor,  1997).  This  project  intends  to  maximize  working  memory  processes  through 
autonomous  intelligent  interruption  and  negotiation  strategy. 

Information-processing  limitations  are  often  described  as  processing  bottlenecks.  The  extent 
of  a  person’s  mental  capacities  and  limitations  remains  a  controversial  matter  (Schumaker 
et  ah,  2001).  Many  researchers  are  currently  testing  theories  regarding  these  cognitive  mental 
capacities.  For  example,  a  common  hypothesis  is  that  a  central  cognitive  decision  and  response 
bottleneck  (RSB)  exists.  The  RSB  is  considered  the  basis  for  explaining  the  limited  human 
capacity  to  process  information  and  is  based  on  the  significant  body  of  literature  that  cites 
severe  and  persistent  limits  found  with  studies  of  dual  (competing)  task  perfonnance  (Pashler, 
1994).  The  argument  is  that  human  processing  is  essentially  serial,  and  response  to  a  secondary 
task  cannot  proceed  until  after  response  to  the  first  stimulus  has  been  completed.  Additional 
theories  are  based  on  the  adaptive  executive  control  (AEC)  model  for  dual-task  performance 
(Meyer  and  Kieras,  1997a,  1997b,  and  1999).  The  AEC  model  indicates  a  declarative  know¬ 
ledge  (i.e.,  verbal  descriptions  about  task  requirements)  is  converted  to  procedural  knowledge 
(Anderson,  1982;  Bovair  and  Kieras,  1991).  Once  the  information  is  converted,  two  tasks  may 
be  performed  simultaneously  in  parallel  by  a  notional  central  processor,  and  thus  the  limiting 
factor  in  processing  is  in  the  sensory  input  processing,  vice  executive  processing  per  se. 

Wickens  (1984)  offered  yet  another  hypothesis  regarding  the  serial  versus  parallel  nature 
of  processing  by  noting  that  there  appeared  to  be  a  pattern  across  the  literature  suggesting 
that  there  were  two  primary  channels  of  processing:  audio-verbal  and  visual-spatial.  Cognitive 
processing  that  was  verbal  and  presented  through  the  auditory  modality  tended  to  be  processed 
in  parallel  to  information  that  was  spatial  in  nature  and  presented  visually.  Thus,  in  effect, 
two  relatively  independent  channels  for  information  processing  exist.  Only  when  information 
supporting  two  different  tasks  was  presented  in  the  same  channel  at  the  same  time  is  interfer¬ 
ence  significant. 

The  notion  of  a  Sensory  Input  bottleneck  refers  to  the  sensing  and  perceiving  of  information 
that  transforms  energy  into  internal  representations  that  can  be  acted  upon  by  cognitive 
processes.  Potential  Sensory  Input  bottlenecks  can  interfere  with  an  individual’s  ability 
to  detect,  identify,  and  properly  classify  targets  for  situation  awareness  (Pew  and  Mavor,  1997). 
These  types  of  interferences  are  a  main  concern  to  an  operator  monitoring  visual  displays  such 
as  radar,  sonar,  and  air  traffic  control.  The  current  objective  is  to  exploit  multiple  sensory 
channels  via  an  autonomous  information  delivery  strategy  to  multiple  modalities. 

Finally,  attention  has  a  long  standing  interest  in  psychology.  One  long  standing  theory 
of  attention  argues  that  perception  is  actively  filtered  by  cognitive  processes,  and  therefore 
attention  can  be  directed.  The  ability  to  attend  to  specific  information  may  be  a  skill 
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that  can  be  trained,  or  may  be  facilitated  by  incorporating  distinctive  features  that  allow  priority 
information  to  be  more  readily  detected. 

Physiological  Monitoring  of  Cognitive  Activity.  The  technologies  developed  over  the  last 
decade  in  measuring  brain  activity  and  various  facets  of  cognition  are  the  foundation  for  the 
development  of  augmented  cognition.  Techniques  like  functional  magnetic  resonance  imaging 
(fMRI)  have  given  scientists  the  opportunity  to  identify  three-dimensional  regions  of  human 
brain  activity  during  specific  mental  tasks.  This  opportunity  opened  up  the  field  of  cognitive 
psychology  substantially  (into  the  new  field  of  cognitive  neuroscience)  and  enabled  researchers 
to  test  their  theories  of  the  human  and  associate  previously  observed  human  behaviors  with 
neural  activity  in  specific  brain  regions.  For  example,  Jiang  (2003)  found  significant  behavioral 
interference  when  participants  must  perform  two  tasks  overlapping  in  time  using  fMRI 
as  a  metric.  The  duration  of  response  to  both  tasks  is  dramatically  increased.  The  physiological 
measures  used  during  Phase  II  of  the  program  will  dynamically  identify  these  types  of  changes 
in  human  cognitive  activity  as  warfighters  are  engaged  in  cognitive  tasks. 
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2.  TRANSITION  AREA  REPORT 


2.1  U.S.  ARMY— FUTURE  FORCE  WARRIOR  (FFW)  (DISMOUNTED  INFANTRY) 

2.1.1  Concept  Validation  Experiment  (CVE)  Analysis  Report  Summary 

Table  7.  Honeywell®  Executive  Summary  of  approach  and  results. 

Concept  Development  and  Analysis  Team  Prime:  Honeywell"  Laboratories 

Application:  Future  Force  Warrior  (FFW) 

Hard  Problems: 

•  Soldiers  are  faced  with  highly  dynamic  task  environments,  many  sources  of  information, 
unpredictable/variable  delivery  times — information  priorities  change  as  context  changes 

•  With  the  advent  of  the  Future  Force  Warrior,  the  task  environment  demands  will  increase  further  due  to 
netted  communications  moving  real-time  information  to  lower  echelons  of  command 

•  Two  aspects  of  attention  are  particularly  salient  to  the  FFW:  capacity  and  allocation 

•  Cognitive  research  indicates  attention  is  a  limited  resource  that  has  a  fluctuating  capacity — available 
resources  are  allocated  through  active  cognitive  processing. 

•  Successfully  augmenting  Attention  bottlenecks  requires  gauges  that  measure  attention  resource  capacity 
and  attention  allocation  of  the  available  resources 

•  The  rigorous  demands  of  FFW  require  sensors  and  digital  signal-processing  technology  adapted  to  size, 

_ weight,  power,  ease  of  use,  and  durability  for  the  soldier  in  the  field _ 

Transition: 

•  Target  Program:  Future  Force  Warrior  Program;  Cynthia  Blackwell  Fluman  Performance  Lead,  Technology 
Program  Office,  insert  technology  by  Sept  2007  for  Land  Warrior  Block  Upgrade  III 

•  FFW  technology  demonstration  has  identified  system  criteria  where  there  must  be  sufficient  increase  in 
survivability/lethality  to  justify  increase  in  additional  size,  weight,  power,  cost,  and  complexity  of  FFW 

_ systems _ 

Test  Conditions: 

IHMC 

•  Simulation-based  (participants  seated)  Military  Operations  on  Urban  Terrain 
(MOUT)  addresses  Attention  bottlenecks 

•  Scenario — Navigate  to  objective 
o  SA  where  they  are,  where  enemy  is,  where  they  need  to  go 
o  Manage  communications 
o  Avoid  ambushes 
o  Engage  enemy 
o  Complete  MEDEVAC 

CMU 

•  Virtual  Reality  (VR)  Simulation  (upright  and  mobile)  MOUT  addresses 
Working  Memory  bottleneck  and  sensor  stability  issues  with  physically  active 
participants 

•  Scenario — Situational  Awareness 
o  Identify  friend/foe 
o  Engage  enemy 

•  Manage  communications _ _ 

IHMC  I  CMU 

•  N  =  13  •  n  =  10 

•  University  students  and  staff  •  University  students 


Scenario 


Participants 
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Table  7.  Honeywell "Executive  Summary  of  approach  and  results,  (cont) 


Gauges 

Gauge  and  Developer: 

Tested 

Used 

Engagement  Index — NASA/Honeywell  R/CCNY 

✓ 

✓ 

Cognitive  Workload  (Stress) — Institute  of  Human 
Machine  &  Cognition 

V 

✓ 

Arousal  Meter — Clemson  University 

s 

✓ 

executive  Load  Index — Human  Bionics 

V 

✓ 

P300  Novelty  Detector — City  College  of  New 
York/Columbia 

V 

✓ 

Integration  Architecture 

•  Sensor  (EEG,  IBI,  HFQRS  ECG,  Pupillometry,  GSR,  Pleth)  output  through 
network  to  agent-based  architecture 

•  Sensor  data  subjected  to  different  analysis  to  support  different  gauge(s) 

•  For  attention  resource  gauges  (sensors  IBI,  ECG,  Pupil,  GSR)  spectral  analysis 
and  linear  regression  on  weighted  means  to  values  for  Stress  and  Arousal 
gauges 

•  For  attention  allocation  gauges  (sensor  EEG)  spectral  analysis  is  used  to  create 
a  XLI,  P300,  Engagement  gauge  values 

•  Output  of  gauges  feeds  rule-based  cognitive  state  profiler  through  agent-based 
architecture 

•  Cognitive  state  profiler  determines  1 )  attention  resources  capacity  2) 
appropriate  allocation  of  attention  resources  to  process  information 

Mitigation  Rationale 

•  Multiple  mitigation  strategies  were  explored  across  CVEs. 

•  Mitigation  strategies  were  context  dependent  and  included 

o  IHMC:  communications  scheduler,  tactile  cueing,  MEDEVAC  agent, 
mixed  initiative  target  id 
o  CMU:  communications  scheduler 

•  Cognitive  state  defined  by  rule  sets  using  different  sets  of  gauges 

o  IHMC:  Any  single  gauge  triggered  the  mitigation 
o  CMU:  At  least  2  of  3  gauges  needed  to  reflect  high  workload  or  rate  of 
change  toward  high  workload  state 

•  Communications  Scheduler 

o  IHMC:  Scheduled  and  presented  messages  based  on  the  cognitive  state 
profile,  the  message  priority,  and  current  context 
o  CMU:  rescheduled  communications  during  high  workload  states 

•  Tactile  Navigation  Cueing 

o  24-element  tactor  belt  (15°  resolution)  provided  pulse  in  desired 
navigation  direction  -  frequency  indicated  proximity 
o  Turned  on  when  attentional  resources  were  limited,  once  triggered  stayed 
on  for  duration  of  navigation  task 

•  MEDEVAC  Agent  -  Executed  medical  evacuation  of  casualties  with  minimal 
information.  Agent  triggered  when  attentional  resources  were  limited. 

Experimental  Design 

•  IHMC:  Single  factor  (mitigation  on/off)  repeated  measures 

•  CMU:  Single  factor  (mitigation  on/off)  repeated  measures 

Independent  Variables 

•  IHMC:  Mitigation  (none,  augmented) 

•  CMU:  Mitigation  (none.,  augmented) 

Dependent  Variables 

•  IHMC 

o  Percentage  change  in  number  of  ambushes  avoided  served  as  measure  of 
attention  resource  capacity 

o  Percentage  of  messages  comprehended  served  as  measure  of  attention 
allocation 

•  CMU 

o  Percentage  of  correctly  recalled  friendly/enemy/ammunition  counts 
served  as  measure  of  working  memory 
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Table  7.  Honeywell  Executive  Summary  of  approach  and  results,  (cont) 


Bottleneck 

Bottleneck 

Goal 

Sample 

Tested 

Team  Results 
(by  Task) 

Mitigation 

Practical 

Demo 

Statistically 

Significant 

Attention 

Allocation: 

100% 

n=  13 

Message 
Comprehension 
(correct  response 
to  msg) 

100% 

Comms 

Scheduler 

IHMC 

p  <  .05 

Attention 

Capacity: 

100% 

n=  13 

%  increase  in 
number  of 
ambushes  avoided 
during  navigation 

380% 

Tactile 

Navigation 

Cueing 

IHMC 

p  <  .01 

Working 

Memory 

100% 

n=  10 

%  correct  of 
memory  for  counts 

155% 

Comms 

Scheduler 

CMU 

p  <  .04 

2.1.2  Background 

Honeywell'  Laboratories  was  selected  to  work  on  IWIIUS  for  the  dismounted  infantry  domain. 
Table  7  is  a  summary  of  the  approach  and  results  to  the  FFW  experiment  conducted  by  the 
Honeywell ®  team.  The  team  conducted  two  CVEs,  one  at  the  Institute  of  Human  and  Machine 
Cognition  (IHMC),  and  the  other  at  Carnegie  Mellon  University  (CMU).  The  IHMC  focus  was  on 
defining  and  measuring  cognitive  bottlenecks  in  the  anticipated  task  environment  for  the  FFW. 
CMU’s  focus  was  on  addressing  the  unknown  challenges  of  trying  to  measure  cognitive  state 
changes  in  task  environments  where  physical  activity  will  be  extensive.  The  primary  bottleneck 
addressed  by  Honeywell8  was  the  Attention  bottleneck  (Note:  the  team  also  performed  experiments 
addressing  the  Working  Memory  bottleneck  at  CMU).  The  attention  resource  and  attention  allocation 
metrics  were  tested  at  IHMC.  Five  gauges  and  associated  biosensors  were  used  to  determine  the 
attention  resource  and  attention  allocation  gauge  values  (see  Table  7  for  complete  listing). 

Development  and  Analysis  Team 

The  Honeywell*  Laboratory  team  included  CMU,  City  College  of  NY,  Clemson  University, 
Columbia  University,  Human  Bionics,  IHMC,  Oregon  Health  and  Sciences  University,  and  UFI. 

2.1.3  Operational  Application — Future  Force  Warrior  (FFW) 

The  team  developed  the  JHAAS  based  upon  operational  enviromnents  expected  of  the  U.S. 

Army’s  FFW  program.  FFW  is  the  Army’s  flagship  Science  and  Technology  initiative  to  develop 
and  demonstrate  how  revolutionary  technology  capabilities  will  transform  the  roles  for,  and 
capabilities  of  future  dismounted  infantry  soldiers.  FFW  notional  concepts  seek  to  create  a 
lightweight,  overwhelmingly  lethal,  fully  integrated  individual  combat  system,  including  weapon, 
head-to-toe  individual  protection,  netted  communications,  soldier-worn  electrical  power  sources,  and 
enhanced  human  physiological  and  cognitive  performance.  The  program  is  aimed  at  providing 
unsurpassed  individual  and  squad  lethality,  survivability,  communications,  and  responsiveness — 
creating  a  formidable  warrior  in  an  invincible  team.  The  FFW  is  a  major  pillar  of  the  Future  Force 
strategy,  complementing  the  Future  Combat  Systems  (FCS)  program. 

Relevance  to  FFW  Application 

FFW  has  a  vision  for  Sensors  and  Communications:  a  “netted”  soldier,  small  units  and  teams  with 
robust  communications,  state-of-the-art  distributed  and  fused  sensors,  organic  tactical  intelligence 
collection  assets,  enhanced  situational  understanding,  on-the-move  planning,  and  linkage  to  other 
force  assets.  Honeywell  "  Laboratories  addresses  these  challenges  with  components  of  the  JHAAS, 
specifically,  the  Communications  Scheduler,  Tactile  Navigation  Cueing,  and  the  MEDEVAC  Agent. 
The  biosensors  used  to  drive  these  cognitive-based  systems  will  also  feed  another  FFW 
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requirement — the  onboard  physiological/medical  sensor  suite  with  enhanced  prompt  casualty  care 
that  is  the  FFW  Mobility  Sustainability  and  Human  Performance  vision.  The  FFW  is  a  major  pillar 
of  the  Future  Force  strategy,  complementing  the  FCS  program. 

“Hard  Problems”  for  Dismounted  Infantry 

Soldiers  are  faced  with  highly  dynamic  task  and  operating  environments — many  sources  of 
information  and  various  contexts  that  have  unpredictable  delivery  times.  Information  priorities 
change  as  context  changes.  With  the  advent  of  the  FFW,  the  task  environment  demands  increase 
dramatically  due  to  netted  communications  moving  real-time  information  through  all  echelons  of 
command.  This  requires  the  FFW  to  perform  the  physical  tasks  expected  of  a  warfighter  today 
(navigate,  communicate,  identify,  and  engage),  but  also  to  process  the  ever-increasing  amount  of  data 
provided  by  netted  communications  and  sensors  to  ensure  situational  awareness  for  self  as  well  as 
provide  situational  awareness  to  others  up  and  down  the  chain  of  command.  Additional  information 
provided  by  netted  communications  is  obviously  valuable,  but  not  at  the  cost  of  the  soldier’s  ability 
to  digest  the  data.  As  discussed  in  Table  7,  cognitive  research  indicates  that  attention  is  a  limited 
resource.  In  the  FFW,  with  competing  requirements  for  those  attentional  resources  increasing 
because  of  the  increasing  data  processing  demands  in  a  netted  communications  environment,  atten¬ 
tion  is  a  resource  that  requires  managing.  The  Honeywell R  team  developed  and  tested  a  system  to 
address  the  Attention  “bottleneck”  problem. 

Transition  Customer 

Honeywell  "  is  part  of  the  integrated  “system-of-systems”  approach  taken  to  support  the  Army 
transformation  to  a  soldier-centric  force,  making  them  well  positioned  to  incorporate  IWIIUS 
technologies  into  the  FFW  program.  The  Honeywell R  team  has  been  working  consistently  with  the 
FFW  Program  and,  in  particular,  the  Human  Performance  Lead,  Technology  Program  Office,  and 
Land  Warrior  Block  III  Upgrade.  Land  Warrior  is  the  Advanced  Concept  Technology  Demonstration 
(ACTD)  that  the  technology  is  targeted  to  transition  to  in  the  beginning  of  Fiscal  Year  (FY)  2008. 

The  industry  lead  on  the  FFW  program,  General  Dynamics  Robotic  Systems  (GDRS)  in  conjunction 
with  the  Natick  Soldier  Center,  will  select  the  IWIIUS  technologies  that  are  mature  enough  to 
transition  to  Land  Warrior.  Technology  insertion  for  FFW  has  a  requirement  to  be  identified  by 
September  2007.  The  transition  customer  has  identified  a  general  metric  of  increased  survivability 
and  lethality  measured  against  increased  size,  weight,  power,  cost,  and  complexity  of  any  technology 
insertion. 

Test  Conditions 

Figure  3  shows  the  general  architecture  used  for  the  IHMC  experiment  (Attention  bottleneck). 
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Figure  3.  Team  Honeywell®  FFW  CLIP  architecture. 

2.1.4  Methodology— IHMC 

The  test  conditions  for  the  Honeywell R  team  consisted  of  a  simulation-based  system  (see  Figure  3) 
that  required  participating  in  a  “first-person  shooter”  scenario  in  a  Military  Operations  on  Urban 
Terrain  (MOUT)  environment.  The  basic  task  required  the  acting  FFW  to  navigate  to  a  pre-assigned 
objective.  During  this  navigation  mission,  the  participant  was  required  to  maintain  situational 
awareness  on  enemy  and  friendly  forces,  manage  communications  with  squad  and  upper  echelons  of 
command,  avoid  ambushes,  complete  a  MEDEVAC,  and  engage  the  enemy  when  confronted. 
Participants  were  volunteer  university  students  and  staff.  They  had  varying  degrees  of  experience  in 
PC  gaming,  but  all  had  the  same  level  of  training  on  this  task  and  had  to  meet  a  criterion  level  of 
performance  before  being  used  as  an  official  test  participant.  The  cognitive  bottlenecks  investigated 
during  this  set  of  experiments  were  attention  resources  and  attention  capacity. 

Cognitive  Bottleneck-Attention  Capacity  and  Attention  Allocation 

Cognitive  research  indicates  attention  is  a  limited  resource  that  has  a  fluctuating  capacity.  Available 
resources  are  allocated  through  active  cognitive  processing.  Successfully  augmenting  Attention 
bottlenecks  requires  gauges  that  measure  attention  resource  capacity  and  attention  allocation  of  the 
available  resources.  Attention  resource  capacity  was  measured  with  an  Inter-beat  Interval  (IB I) 
monitor,  ECG,  pupillometry,  and  GSR  sensors.  Attention  allocation  was  measured  with  an  EEG 
sensor.  These  sensors  provided  raw  input  that  was  processed  to  provide  five  gauges  of  cognitive 
state.  The  values  of  these  gauges  were  then  subjected  to  a  rules-based  system  that  determined  the 
overall  state  of  attention  resources  and  attention  allocation.  Depending  on  the  output  of  this  rules- 
based  system,  the  different  mitigation  strategies  were  used  to  enhance  FFW  performance. 
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Mitigation 

Three  types  of  mitigation  were  used  to  improve  FFW  performance.  A  Communications  Scheduler 
was  used  to  schedule  and  present  messages  based  on  the  cognitive  state  profile,  the  message  priority, 
and  current  context.  Essentially,  when  the  cognitive  state  profile  indicated  attention  allocation  was 
saturated  (could  not  pay  attention  because  of  other  attention  priorities  or  overload),  communications 
would  be  delayed  until  such  time  as  attention  resources  were  again  available.  A  second  mitigation, 
Tactile  Navigation  Cueing,  used  a  24-element  tactor  belt  (15°  resolution)  and  provided  a  pulse 
in  desired  navigation  direction  while  frequency  of  pulses  indicated  proximity  to  objective.  This 
mitigaion  was  turned  on  when  attention  capacity  was  limited.  Finally,  a  MEDEVAC  Agent  mitigation 
executed  medical  evacuation  of  casualties  with  minimal  information  required  from  the  FFW. 

The  Agent  was  triggered  when  attention  capacity  was  limited. 

Independent  and  Dependent  Variables 

As  noted  in  Table  1,  this  experiment  used  a  single-factor  repeated  measures  experimental  design. 
The  independent  variable  was  Mitigation  ON  or  Mitigation  OFF.  The  dependent  variable  was  the 
number  of  ambushes  avoided.  If  the  participant  had  sufficient  attentional  capacity,  communicated 
warnings  were  processed  and  correct  navigation  was  accomplished,  allowing  for  minimum  contact 
with  enemy  ambushes.  If,  however,  capacity  was  insufficient,  the  participant  would  miss  warnings 
and/or  navigate  into  areas  that  were  off  the  correct  course  to  objective,  significantly  increasing 
encounters  with  enemy  ambushes.  The  number  of  correct  responses  to  queries  was  measured  for 
the  attention  allocation-dependent  variable.  In  this  case,  the  participants’  attention  allocation  may 
be  saturated  (paying  attention  to  higher  priorities  or  overloaded  with  too  many  things  to  pay  attention 
to),  stressing  a  decrement  in  their  ability  to  respond  to  queries. 

2.1.5  Methodology — CMU 

In  a  second  set  of  experiments  conducted  by  the  Honeywell1'  team,  a  more  immersive  environment 
was  used.  In  this  set  of  tests,  a  Virtual  Reality  (VR)  system  provided  visualization  through  a  set 
of  head-mounted  goggles  and  tracked  the  actual  movement  of  the  participant.  This  was  the  first  step 
toward  moving  into  a  real  “field”  experiment  where  the  sensors  mounted  on  the  FFW  will  be  subject¬ 
ed  to  movement,  shock,  and  other  environmental  extremes.  The  scenario  required  the  participant 
to  operate  from  a  rooftop  location  and  monitor  surrounding  buildings.  The  participant  was  required 
to  identify  and  count  friendly  and  enemy  forces,  eliminate  enemy  forces,  and  concurrently  manage 
communications  (report  different  counts  to  higher  command).  The  cognitive  bottleneck  investigated 
with  this  experiment  was  working  memory.  The  participants  again  were  university  students  and  had 
to  meet  a  minimum  performance  level  through  training  before  being  an  official  test  participant. 

Independent  and  Dependent  Variables 

As  noted  in  Table  1,  this  experiment  used  a  single-factor  repeated  measures  experimental  design. 
The  independent  variable  was  also  Mitigation  ON  or  Mitigation  OFF.  The  dependent  variable 
was  percent  correctly  recalled  of  friendly/enemy/ammunition  counts.  The  better  the  recall,  the  better 
working  memory  was  functioning.  If  working  memory  had  sufficient  capacity  and  could  process 
the  numbers  counted  (e.g.,  interference  in  working  memory  would  be  using  the  same  cognitive  areas 
to  manage  counts  and  to  answer  a  communications  query — causing  a  capacity  and  processing 
conflict),  the  participant  could  easily  keep  track  of  counts  and  recall  them  when  reporting  to  higher 
command.  However,  if  the  participant  was  in  a  conflicting  working  memory  situation  (see  example 
in  previous  sentence)  between  counting  and  processing  communications,  both  tasks  suffered. 
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Cognitive  Bottleneck — Working  Memory 

Working  memory  is  that  part  of  our  cognitive  system  that  we  are  consciously  aware  of  at  any  given 
time.  Working  memory  can  be  thought  of  as  analogous  to  random  access  memory  (RAM) 
on  a  computer.  We  have  lots  of  memory  storage  capacity  in  our  cognitive  system,  but  the  vast 
majority  of  that  memory  is  permanent  memory  (analogous  to  hard-disk  storage  on  a  computer). 

When  we  communicate,  we  have  to  get  a  meaningful  unit  of  speech  or  text  into  working  memory 
to  interpret  it.  The  problem  with  slow  or  inaccurate  communication  is  that  working  memory  can  only 
hold  information  for  about  10  seconds  and  then  it  decays.  If  communication  is  delayed  or  requires 
considerable  processing,  by  the  time  you  get  the  last  word  of  a  meaningful  dialog  into  working 
memory,  the  first  words  may  have  decayed.  Messages  would  need  to  be  repeated,  making 
communications  slow  and  laborious,  and  comprehension  very  difficult. 

Mitigation 

This  set  of  experiments  used  one  mitigation  strategy.  The  Communications  Scheduler  rescheduled 
communications  during  high-working-memory  workload  states. 

2.1.6  Results 

The  DARPA  goal  was  to  achieve  a  1 00%  improvement  in  the  cognitive  Attention  bottleneck. 

The  results  obtained  were  statistically  significant  at  the  p  <  0.05  level,  and  were  done  within  system 
performance  constraints  specified  at  the  start  of  IWIIUS  Phase  II.  As  can  be  seen  in  Table  7,  this  goal 
was  matched  or  surpassed  for  those  bottlenecks  addressed  by  the  Honeywell®  team.  In  the  Attention 
Allocation  bottleneck  (measured  by  message  comprehension),  a  100%  improvement  was  observed, 
meaning  that  the  FFW’s  ability  to  process  communications  while  in  a  combat  environment  doubled. 
This  has  the  potential  effect  of  allowing  improved  situational  awareness  and  therefore  increased 
survivability  and  more  effective  force  on  force  engagements.  In  the  Attention  Capacity  bottleneck 
(measured  by  the  number  of  avoided  ambushes),  a  380%  improvement  was  observed,  meaning  that 
nearly  four  times  as  many  ambushes  were  avoided  as  was  found  with  the  baseline  condition  due  to 
the  availability  of  the  AugCog.  This  increase  can  be  directly  related  to  survivability  of  the  FFW — 
with  AugCog  mitigation,  the  FFW  will  have  better  situational  awareness,  cognitively  process  those 
tasks  that  have  direct  bearing  on  survivability  and  lethality,  and  improve  task  performance  times  and 
accuracy. 

2.1.7  Phase  III  Goals 

Phase  III  will  provide  more  challenges  as  IWIIUS  technology  is  applied  to  real-world  applications 
and  under  conditions  of  operational  stress  (in  the  field).  Some  challenges  include  the  following: 

•  Under  operational  conditions  of  cognitive,  physiological,  and  environmental  stress,  mitigate  the 
impact  of  platform-specific  stressors  on  cognition  and  information  processing  performance 

•  Real-time  signal  processing  (artifact  detection) — motion  induces  noise  (artifacts)  that  makes 
cognitive  state  detection  more  difficult 

•  Context  modeling  with  new  field  sensors — need  sensors  to  understand  the  environment  where 
decisions  are  made  (in  the  field) 

•  Real-time  cognitive  state  classification  through  neural  networks  (NN) — need  to  develop  NNs 
tuned  to  individuals  performing  specific  (identifiable)  tasks 

Probably  more  significant  will  be  the  transition  from  the  laboratory  environment  to  the  field,  which 
will  include  going  to  U.S.  Army  personnel  in  evaluations  and  experiments.  System  metrics  will  be 
developed  to  support  FFW  objectives  (survivability/lethality)  and  performance  on  task  (better 
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situation  awareness,  faster  completion  of  task,  improved  survivability,  increased  lethality)  will  be  the 
metric  challenges  in  Phase  III.  A  technology  metric  will  be  reliable  classifications  (90%  correct 
classification  over  blocks  of  time)  in  a  “field”  environment. 

More  key  to  the  success  of  Phase  III  and  the  future  use  of  IWIIUS  technology  will  be  user 
acceptance.  The  FFW  goals  are  clear — survivability  and  lethality  must  be  enhanced  for  any 
technology  to  be  acceptable  to  the  future  warfighter.  Phase  II  has  shown  that  the  promise  is  there; 
Phase  III  will  have  the  challenge  of  taking  the  next  step  toward  warfighter  employment. 


32 


2.2  U.S.  MARINE  CORPS— MARINE  AIR  GROUND  TASK  FORCE 
(MAGTF)  LIGHT  ARMORED  VEHICLE  (LAV) 

2.2.1  Concept  Validation  Experiment  (CVE)  Analysis  Report  Summary 

Table  8.  DaimlerChrysler®  Executive  Summary  of  approach  and  results. 

Concept  Development  and  Analysis  Team  Prime:  DaimlerChrysler  Corporation  (DCC) 
Application:  MAGTF  Light  Armored  Vehicle  (LAV) 

Hard  Problems: 

•  Multi-tasking  under  high  stress:  Move,  shoot,  and  communicate  (performed  all  simultaneously  by  the  crew 
members) 

•  Sensory  input  bottlenecks  operationally  defined  in  terms  of  tasks  that  compete  for  limited  sensory  processing 
resources.  Driving  requires  high  levels  of  visual  processing — need  to  present  additional  tasks  through  alternate 
modalities,  e.g.,  auditory.  Need  to  validate 

•  Cognitive  bottlenecks  may  also  be  task-specific — must  operationally  define  and  validate  in  operational  context 

_ to  predict  if  augmentation  is  going  to  be  effective _ 

Transition: 

•  Target  Program:  MEFFV:  Marc  Miller,  USMC  Vehicles  Branch — Code  2120,  NSWC  Carderock _ 

Test  Conditions: 


Scenario 

•  Driving  under  real  conditions  on  German  highways 

•  Tasks  create  high  demands  on  working  memory,  auditory  and  visual  sensory 
processing 

o  Primary  task  was  to  drive  (high  demands  on  visual  processing  and  working 
memory) 

o  Secondary  tasks  include  listening  to  situational  brief  and  questions, 
performing  navigational  math,  executing  driving  maneuvers,  acknowledging 
auditory  or  visual  commands  (creating  high  demands  for  auditory  processing, 
working  memory,  visual  processing) 

Participants 

•  N=  5 

•  Male,  right-handed,  DCC  test  drivers,  average  age  of  30  years 

Gauges 

Gauge  and  Developer: 

Tested: 

Used: 

EEG:  auditory  sensory  input  gauge 
(FIRST,  Berlin,  Germany) 

✓ 

✓ 

EEG:  working  memory  gauge 
(FIRST,  Berlin,  Germany) 

✓ 

✓ 

Context  +  driving  behavior  +  seat  posture: 

visual  sensory  input  gauge 

(Sandia  Ntl.  Lab,  Albuquerque,  NM; 

University  of  Pittsburgh,  PA) 

✓ 

✓ 

Integration  Architecture 

•  Sensors  connected  to  stand-alone  gauge  processors 

•  Gauge  outputs  shared  over  network  through  vehicle’s  CAN  data-bus  technology 

•  Cognitive  states  based  on  specific  task  demands.  Linear  Discriminate  Analysis 
based  on  EEG  data  and  neural  network  analysis  based  on  context  data  are  used  to 
define  high/low  states 

•  EEG  classifiers  finalized  during  training  session  for  each  participant 

Mitigation  Rationale 

•  Modality  switch  -  Auditory  to  Visual.  This  strategy  was  used  when  the  Driving- 
Maneuvers  task  was  performed  (implying  that  auditory  modality  was  available) 

•  Scheduling  -  Delayed  presentation  of  information.  This  strategy  was  used  when 
Situational-Brief  or  Navigational-Math  tasks  were  performed,  i.e.,  when  visual  and 
auditory  modality  were  at  capacity 
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Table  8.  DaimlerChrysler®  Executive  Summary  of  approach  and  results,  (cont) 


Experimental  Design 


Each  participant  performed  one  Baseline  Session  (without  mitigation  system)  and 
one  AugCog  Session  (with  mitigation  system),  2  hours  each 
Single-factor  (mitigation  on/off)  repeated  measures 


Independent  Variables 


Mitigation  (none,  augmented) 


Dependent  Variables 


Information-processing  performance  (processing  speed  and  accuracy)  on  competing 
secondary  task  (auditory  or  visual  commands,  respectively) 


Bottleneck 

Bottleneck 

Goal 

Sample 

Tested 

Team  Results 
(by  Task) 

Mitigation 

Practical 

Demo 

Statistically 

Significant 

Sensory  Input 
(Auditory) 

100% 

N=  5 

auditory 

commands 

108% 

scheduling 

V 

one-tailed  Wilcoxon 

Sensory  Input 
(Visual) 

100% 

N=  5 

visual 

commands 

72% 

modality 

switch 

S 

(paired  rank-sum) 
test: 

Working 

Memory 

500% 

N=  5 

auditory 

commands 

103% 

scheduling 

V 

T  (15)  =  27,  p  <  0.05 

2.2.2  Background 

DaimlerChrysler  Corporation  (DCC)  was  selected  to  work  on  IWIIUS  for  the  combat  vehicle 
domain.  Table  8  is  a  summary  of  the  approach  and  results  of  manipulating  the  cognitive  state 
of  an  active  driver.  The  environment  and  tasks  were  structured  to  replicate  a  military  environment 
while  maintaining  experimental  controls  (e.g.,  test  participants,  vehicle  instrumentation,  and 
repeatability  of  driving  environment).  The  primary  cognitive  bottleneck  assigned  to  DCC  was  the 
Sensory  Input  bottleneck.  Three  gauges  and  associated  sensors  were  used  to  determine  the  cognitive 
loading  (general  workload),  sensory  inputs  (verbal  or  spatial),  and  task  context  (turning,  changing 
lanes,  etc.) 

Development  and  Analysis  Team 

The  DCC  team  included  DaimlerChrysler  AG,  University  of  Pittsburgh,  Fraunhofer  Institute 
FIRST,  Sandia  National  Laboratories,  and  Transferzentrum  fur  Mikroelektronik. 

2.2.3  Operational  Application — Light  Armored  Vehicle  (LAV) 

The  MEFFV  is  a  combat  vehicle  replacement  project  designed  to  provide  the  capabilities  presently 
conferred  by  the  LAV  family  and  the  M1A1  Main  Battle  Tank.  It  is  envisioned  that  this  family 
of  vehicles  will  facilitate  and  enhance  the  capabilities  of  Expeditionary  Maneuver  Warfare  through 
technologies  to  significantly  improve  efficiency  and  lessen  the  impact  of  operational  and  tactical 
logistics.  The  family  of  vehicles  will  be  designed  to  maximize  deployability  options  while  retaining 
the  requisite  combat  capabilities  (lethality,  survivability,  mobility,  sustainability)  to  perform  the  roles 
of  the  vehicles  replaced.  MEFFV  should  replace  the  LAV  and  tank  fleets  when  they  reach  their 
respective  end-of-service  dates  (2015  and  2020,  respectively). 

Relevance  to  MEFFV  Application 

As  stated  above,  MEFFV  will  replace  the  United  States  Marine  Corps’  (USMC)  current  family 
of  LAVs  and  the  M1A1  Main  Battle  Tank.  However,  the  intent  is  not  a  one  for  one  replacement; 
the  expectation  is  a  modular  design  concept  that  will  maximize  component  commonality  to  minimize 
operations,  training,  and  maintenance  costs  and  provide  potential  manpower  structure  savings. 
Additionally,  the  MEFFV  will  be  expected  to  take  advantage  of  network-centric  warfare.  Thus, 
the  MEFFV  program  is  faced  with  a  growing  amount  of  available  information  through  advanced 
networked  sensors  and  communications.  An  additional  requirement  is  to  provide  a  reduction 
in  manpower  for  future  vehicles.  The  warfighter  will  have  to  deal  with  the  highly  visual  task 
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of  driving  a  vehicle  while  simultaneously  dealing  with  communications  and  sensor  management. 
Allowing  the  warfighter  to  accomplish  multiple  and  competing  tasks  will  require  the  vehicle  and 
information  interfaces  to  have  driver’s  state  “awareness” — both  context  (e.g.,  maneuvering  around 
obstacles,  avoiding  enemy  contact,  maintaining  formation  with  other  vehicles,  etc.)  and  cognitive 
(e.g.,  processing  spatial  information,  processing  verbal  information). 

“Hard  Problems”  for  MEFFV 

Multi-tasking  under  high  stress — move,  shoot,  and  communicate,  each  performed  simultaneously 
by  the  warfighter — is  the  fundamental  hard  problem  for  the  MEFFV.  Reducing  the  number  of  crew 
members  from  current  manning  exacerbates  the  problem.  The  warfighter  has  a  limited  capacity 
(bottleneck)  for  processing  sensory  input.  Sensory  Input  bottlenecks  are  operationally  defined 
in  terms  of  tasks  that  compete  for  those  limited  sensory  processing  resources.  Driving  requires  high 
levels  of  visual  processing,  depending  on  driving  task  or  context.  Thus,  additional  tasking  (communi¬ 
cations,  sensor  management)  may  need  to  be  presented  through  alternate  modalities,  e.g.,  auditory. 
When  and  how  to  shift  between  modalities  for  competing  tasks  (simultaneous  visual  or  verbal)  and 
the  benefit  of  mitigation  to  operational  performance  is  the  “hard  problem”  that  the  DCC  team 
is  addressing. 

Transition  Customer 

The  DCC  team  is  working  with  the  Office  of  Naval  Research,  the  Marine  Corps  Systems 
Command  MEFFV  program  office,  the  USMC  Vehicles  Branch  (Code  2120),  and  Naval  Surface 
Warfare  Center,  Carderock  Division.  The  Carderock  Division  is  contributing  to  the  development 
of  the  MEFFV  by  looking  at  technologies  to  insert  into  the  MEFFV  program.  The  USMC  Vehicles 
Branch  is  working  with  DCC  to  provide  operational  context  for  further  developing  IWIIUS  technol¬ 
ogies. 

Test  Conditions 

Figure  4  shows  the  general  architecture  used  by  the  DCC  team. 
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Figure  4.  Team  DCC  CLIP  architecture. 


2.2.4  Methodology 

The  test  conditions  for  the  DCC  team  consisted  of  an  instrumented  Mercedes  Benz  R  S  class 
vehicle  driving  on  German  highways  in  and  around  Stuttgart,  Germany.  The  primary  task  was  to 
drive  the  vehicle  safely  and  navigate  between  waypoints.  Competing  tasks  were  then  introduced. 
These  consisted  of  listening  to  a  situation  brief  and  responding  to  questions,  performing  navigational 
mathematics,  executing  directed  driving  maneuvers,  and  acknowledging  auditory  or  visual 
commands.  The  primary  task  provided  a  high  demand  for  visual  sensory  processing.  The  secondary 
tasks  provided  competing  visual  tasks  to  the  primary  tasks  and  provided  competing  tasks  for  auditory 
(or  verbal  sensory  input)  processing. 

Cognitive  Bottleneck — Sensory  Input 

The  DCC  team  addressed  the  Sensory  Input  cognitive  bottleneck.  This  bottleneck  can  be  described 
as  the  cognitive  areas  necessary  for  processing  different  sensory  inputs  such  as  verbal  and  spatial 
inputs.  Cognitive  science  suggests  that  most  processing  areas  of  the  brain  are  synchronous.  Thus, 
if  the  area  responsible  for  processing  verbal  inputs  is  “busy,”  additional  verbal  inputs  will  potentially 
be  missed  or  ignored.  However,  the  brain  can  also  parallel  process;  thus,  if  the  verbal  processing 
areas  are  busy  and  new  verbal  inputs  are  available,  it  is  believed  that  a  modality  switch — changing 
verbal  input  to  spatial  input — would  allow  immediate  processing  of  the  new  verbal  input,  but  in 
another  modality.  Another  method  of  handling  competing  verbal  (or  other  modalities)  inputs  would 
be  to  use  a  scheduling  device  that  would  know  when  verbal  processing  was  “busy”  and  when  it 
was  “available.”  The  DCC  team  used  these  concepts  and  additional  information  from  context  sensors 
to  develop  their  experimental  theories  and  mitigations  to  optimize  the  warfighter/ driver  performance. 


36 


Mitigation 

Table  8  shows  the  mitigation  strategies  used  during  the  CVE  experiment. 

•  Mitigation  1.  Modality  switch  (auditory  to  visual).  This  strategy  was  used  when  the  Driving- 
Maneuvers  task  was  performed  (implying  that  auditory  modality  was  available). 

•  Mitigation  2.  Scheduling  (delay  presentation  of  information).  This  strategy  was  used  when 
Situational-Brief  or  Navigational-Math  tasks  were  performed,  i.e.,  when  visual  and  auditory 
modality  were  at  capacity. 

Mitigation  1  provided  the  capability,  while  the  vehicle  was  maneuvered  (high  spatial  processing 
task),  to  change  additional  spatial  tasks  (competing  with  driver  maneuvering)  to  another  modality, 
in  this  case,  auditory  (verbal-processing)  commands.  Mitigation  2  was  used  when  the  participant 
was  using  normal  driving  skills,  but  given  a  situation  brief  or  asked  to  perform  a  navigational 
mathematical  problem.  While  listening  to  the  situation  brief,  the  participant’s  auditory  or  verbal 
processing  was  at  or  near  capacity.  Introducing  additional  verbal  tasks  would  cause  those  tasks 
to  be  ignored  or  cause  a  decrement  in  performance  on  both  verbal  tasks.  Given  the  known  state 
of  the  participants’  sensory  processing,  a  scheduling  mitigation  technique  was  implemented  for 
verbal  tasks. 

Independent  and  Dependent  Variables 

As  noted  in  Table  8,  this  experiment  used  a  single-factor  repeated  measures  experimental  design. 
The  dependent  variable  was  Mitigation  ON  or  Mitigation  OFF.  The  dependent  variable  was  the 
information-processing  performance  (processing  speed  and  accuracy)  on  a  competing  secondary  task. 
Essentially,  the  participant,  while  maneuvering  the  vehicle  through  normal  highway  traffic,  was 
presented  with  competing  tasks  that  required  spatial  (visual)  or  verbal  (auditory)  processing — how 
well  the  participant  processed  these  competing  tasks  (accuracy)  and  how  quickly  the  participant 
performed  the  tasks  was  the  performance  measurement  for  this  experiment. 

2.2.5  Results 

The  DARPA  goal  was  to  achieve  a  100%  improvement  in  the  cognitive  Sensory  Input  bottleneck. 
The  results  obtained  were  statistically  significant  at  the  p  <  0.05  level  and  were  done  within  system 
performance  constraints  specified  at  the  start  of  Phase  II.  As  can  be  seen  in  Table  8,  operator 
performance  improved  108%  when  the  mitigation  was  activated  in  dealing  with  the  auditory/verbal 
sensory  input.  The  spatial  or  visual  sensory  input  resulted  in  a  72%  increase.  This  result  is  explained 
when  one  considers  that  the  primary  operating  environment  is  driving — highly  visual.  This  environ¬ 
ment  leads  to  the  conclusion  that  spare  capacity  is  limited  in  the  visual/spatial  channel  and  a  72% 
improvement  is  actually  quite  remarkable.  With  the  present  prototype,  DCC  demonstrated  that  it 
is  possible  to  reliably  detect  cognitive  state  changes  and  that  this  information  allows  significant 
improvement  in  operator  performance.  This  capability  should  allow  a  warfighter  to  process  two 
to  three  times  more  information  as  compared  to  a  typical  combat  vehicle  driver  of  today. 

2.2.6  Phase  III  Goals 

The  Phase  III  goal  is  to  mitigate  the  impact  of  platform-specific  stressors  on  cognition  and 
information-processing  performance  while  under  operational  conditions  of  cognitive,  physiological, 
and  environmental  stress.  In  light  of  this  main  goal,  susceptibility  of  EEG-based  gauges  to  artifacts 
caused  by  mechanical  vibrations  or  shocks  must  be  addressed.  In  a  G-Class  (combat)  vehicle  used 
on  offroad  courses,  artifacts  may  be  expected  to  be  of  a  different  nature  than  in  the  current  S-Class 
vehicle.  Thus,  it  may  be  necessary  to  develop  online  artifact  rejection  algorithms  that  allow  sufficient 
cleaning  of  the  data  before  passing  them  to  the  cognitive  state  classifiers.  In  addition,  future 
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recording  techniques  (e.g.,  contact-free  measurement)  may  be  less  susceptible  to  movement, 
vibration,  or  shock  artifacts.  Additionally,  DCC  plans  to  migrate  their  system  to  the  operational 
environment  of  the  MEFFV  in  collaboration  with  the  USMC  Vehicles  Branch.  This  plan  will  require 
adapting  the  operational  environment  tasks  to  USMC  specifications  and  to  adapt  gauges  to  those 
tasks. 

More  significantly,  a  new  challenge  for  Phase  III  will  be  to  simultaneously  monitor  the  cognitive 
states  of  multiple  operators  within  a  team  and  to  develop  significantly  more  sophisticated  mitigation 
algorithms  to  balance  tasking  across  the  whole  team.  The  goal  will  be  to  demonstrate  the  system’s 
capability  of  dynamically  distributing  crew  workload  among  driver,  commander,  and  vehicle,  thus 
increasing  the  crew’s  operational  performance  and  safety. 
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2.3  U.S.  NAVY— TACTICAL  STRIKE  COORDINATOR  (TSC)  FOR  THE  TACTICAL  TOMAHAWK 
WEAPONS  CONTROL  SYSTEM  (TTWCS) 

2.3.1  Concept  Validation  Experiment  (CVE)  Analysis  Report  Summary 


Table  9.  Lockheed  Martin®  Executive  Summary  of  approach  and  results. 

Team:  Lockheed  Martin®  Advanced  Technology  Laboratory  (LMATL) 

Application:  Tactical  Tomahawk — Tactical  Strike  Coordinator  (TSC) 

Hard  Problems: 

•  TSC  anticipated  as  taking  on  new  functions  to  support  real-time  mission  re-targeting.  Will  require  single 
person  to  do  what  three  people  are  currently  doing 

•  Significant  re-planning  components  with  simultaneous  monitoring  of  ongoing  missile  missions  will 
overload  working  memory 

•  Must  remember  missile  capabilities,  potential  track  conflicts,  which  missiles  can  service  which  targets, 

_ variable  time  constraints,  and  long-term  implications  to  future  missions _ 

Transition: 

•  TTWCS — PMA  282,  NAV AIR’s  Cruise  Missile  Weapons  Control  Systems  Program  Office,  Advanced 
Concepts  Group 

•  POC:  LCDR  Eric  LeGear,  Advanced  Concepts,  CAPT  Sullivan,  PMA  282 _ 

Test  Conditions: 


Scenario 

•  TTWCS  desktop  simulation 

•  Task  Demands: 

o  Location  (Spatial  Memory)  Task:  When  augmented  video  display  fails, 
remember  which  missiles  are  striking  which  targets 
o  Alert  (Situation  Awareness)  Task:  Questions  asked  through  text  in  Chat 
window  re:  current  situation 

o  Retarget  (Working  Memory)  Task:  Emergent  target  tracks  appear  on  video 
display.  Participant  must  quickly  process  time  and  capabilities  constraints 
for  numerous  missiles  and  targets 

Participants 

•  N=  12 

•  LMATL  personnel,  not  experts 

Gauges 

Sensor: 

Tested 

Used: 

GSR:  Anthrotronix 

✓ 

V  (data  only) 

EKG:  Anthrotronix 

✓ 

•S  (data  only) 

EEG:  Advanced  Brain  Monitoring 

✓ 

•S  (data  only) 

Pupillometry:  Index  of  Cognitive 
Activity  (ICA)  -  Eye  Tracking, 

Inc. 

✓ 

✓ 

fNIR  -  Drexel  University. 

✓ 

Integration  Architecture 

•  Cognitive  State  Assessor  (CSA)  processor  uses  neural  network  to  identify  two 
gauge  values  (verbal  and  spatial  working  memory — based  on  a  mix  of 
preprocessed  data  and  ICA  gauge  output) 

•  Performance  Augmentation  through  PACE  architecture  is  rule-based  system 
determining  what  augmentation  should  be  triggered  and  when 

Mitigation  Rationale 

•  Intelligent  Sequencing — Change  the  order  of  sequentially  presented  verbal 
and  spatial  tasks 

Experimental  Design 

•  Single-factor  (mitigation  on/off)  repeated  measures 
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Table  9.  Lockheed  Martin®  Executive  Summary  of  approach  and  results,  (cont) 


Independent  Variables 

•  Mitigation  (none,  augmented) 

Dependent  Variables 

•  Performance  score  for  retargeting  task  with/without  augmentation. 

Bottleneck 

Bottleneck 

Goal 

Sample 

Tested 

Team 

Results 

Mitigation 

Practical 

Demo 

Statistically 

Significant 

Working 

Memory 

500% 

n=  12 

642% 

Intelligent  Sequencing 

✓ 

?  (11)  =5.84, 

p<.001 

2.3.2  Background 

The  Lockheed  Martin8  analysis  report  discusses  the  Cognitive  Task  Environment s),  human  state 
gauges,  and  biosensors  used;  research  methods;  integrated  prototype  system  design;  and  empirical 
results.  LMATL  was  the  chosen  performer  to  work  on  IWIIUS  in  the  Tactical  Tomahawk  Weapons 
Control  System  (TTWCS)  domain.  The  LMATL  focus  was  on  the  cognitive  task  environment 
of  the  TSC;  Table  9  is  a  summary  of  the  approach  and  results  of  manipulating  the  cognitive  state 
of  the  operator.  The  primary  bottleneck  addressed  by  LMATL  was  the  Working  Memory  bottleneck. 
Five  gauges  and  associated  biosensors  were  used  to  condense  to  two  gauge  values  (verbal  and  spatial 
working  memory)  through  a  neural  network. 

Development  and  Analysis  Team 

The  Lockheed  Martin®  team  was  led  by  LMATL  and  included  Design  Interactive,  Anthrotronix, 
Advanced  Brain  Monitoring,  Drexel  University,  University  of  Virginia,  and  Eyetracking,  Inc. 

For  Phase  II,  the  team  developed  the  simulated  TTWCS  within  a  CLIP -based  on  operational  environ¬ 
ments  expected  of  the  U.S.  Navy’s  Tactical  Tomahawk  program. 

2.3.3  Operational  Application — TTWCS  Tactical  Strike  Coordinator  (TSC) 

TTWCS  is  the  Navy’s  evolutionary  HCI  development  approach  undertaken  by  the  Advanced 
Concepts  Group  under  the  Cruise  Missile  Weapons  Control  Systems  Program  Office  (PMA-282) 
to  combine  several  interfaces  into  an  integrated  land-attack  presentation  layer.  TTWCS  is  a  fully 
integrated  Joint  Service  partner,  exchanging  information  and  battle  graphics  with  other  service 
systems.  The  new  TTWCS  effort  is  being  integrated  by  Lockheed  Martin8  Integrated  Systems  and 
Solutions  (IS&S).  With  the  introduction  of  the  Tomahawk  Block  IV  upgrade,  new  challenges 
are  being  placed  on  the  planners  and  executers  of  a  Tomahawk  Land-Attack  Missile  (TLAM)  event. 
The  Tactical  Tomahawk  will  use  onboard  mission  planning,  in-flight  retargeting,  and  battle  damage 
assessment  capabilities.  The  goals  for  the  upgraded  TLAM  are  to  service  high-priority,  time-critical 
targets,  with  a  goal  of  less  than  1 0  minutes  from  target  identification  to  target  destruction. 

“Hard  Problem”  for  the  TSC 

The  introduction  of  the  new  Tactical  Tomahawk  means  that  a  layer  of  human  control  will  be  need¬ 
ed  where  none  previously  existed.  The  Tactical  Tomahawk  case  illustrates  a  perplexing  problem 
for  cognitive  systems  engineers  designing  a  supervisory  control  system  to  support  monitoring  and 
resource  allocation. 

The  TSC  has  overall  responsibility  for  TLAM  operations,  determining  mission  and  missile  require¬ 
ments,  which  launch  a  platform  to  task.  In  the  future,  the  TSC  will  be  expected  to  do  the  work  that 
three  operators  currently  perform.  The  new  capabilities  of  retargeting,  re -planning,  and  simultaneous 
monitoring  of  missions  in  flight  will  overload  working  memory  of  a  single  TSC.  Working  memory 
is  a  capacity-limited  and  time-constrained  resource.  The  TSC  must  remember  missile  configuration, 
look  for  potential  track  conflicts,  ensure  proper  missile-to-target  pairing,  and  account  for  temporal 
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constraints  and  the  implications  of  current  missions  on  future  mission  planning.  These  tasks  will 
compete  for  use  of  working  memory.  The  hard  problem  for  the  LMATL  team  is  maximizing  the 
efficiency  of  the  operator’s  working  memory  while  increasing  operational  performance — doing 
the  work  of  three  with  one. 

Transition  Customer 

The  LMATL  Team  has  been  working  with  the  Lockheed  Martin®  IS&S  Division  TTWCS  Program 
internally  and  externally  with  Naval  Air  Systems  Command’s  (NAVAIR)  Advanced  Concept  Group 
under  the  Cruise  Missile  Weapons  Control  Systems  Program  Office  (PMA-282).  The  ability 
to  redirect  an  in-flight  missile,  loiter  a  missile  to  allow  quick  reaction  re-planning,  and  provide  battle 
damage  assessment  has  been  successfully  brought  forward  with  the  Block  IV  Tomahawk  and  through 
the  development  of  TTWCS.  Design  changes  are  planned  for  later  versions  of  TTWCS  to  improve 
the  HCI  (this  is  where  LMATL  comes  in  with  AugCog).  LMATL  cognitive  research  will  improve 
TTWCS  to  achieve  a  reduction  in  operator  workload  through  mitigation  techniques  triggered 
by  a  biosensor-driven  neural  network  in  a  closed  loop,  which  will  ultimately  improve  operator 
performance.  The  transition  customer  has  identified  a  general  metric  of  the  TSC  doing  the  work 
of  three  Tomahawk  operators. 

Test  Conditions 


The  LMATL  prototype  system  is  designed  around  the  Performance  Augmentation  through 
Cognitive  Enhancement  (PACE)  architecture  (Figure  5).  PACE  is  a  highly  reusable  architecture  that 
manages  user  tasks  and  allows  implementing  various  mitigation  strategies  in  a  domain-independent 
fashion.  PACE  consists  of  several  components  that  are  entirely  domain-independent  and  several  that 
are  designed  to  be  extended  for  a  particular  domain. 

Interacting  with  PACE  is  the  prototype  jTTWCS  (Java  Tactical  Tomahawk  Weapon  Control 
System),  which  presents  a  user  interface  allowing  the  operator  to  perform  the  TTWCS  tasks 
of  missile  monitoring  and  retargeting. 


Configuration 
Files 


Task 

Attributes 


Active 

Task 

Manager 

(ATM) 


User 
Performance  I 


Task 

Information 

Manager 

(TIM) 


New 

Tasks 


Cognitive 

State 
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(CSA) 


Cognitive 

State 


System 

Interface 

Director 

(SID) 


Proposed 

Tasks 


Adaptive 

Workload 

Director 
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Delegated 

Tasks 


Delegation 

Manager 

(DM) 


Figure  5.  PACE  architecture — major  components  and  key  interactions. 
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2.3.4  Methodology 

The  test  conditions  for  the  LMATL  team  consisted  of  a  desktop  TTWCS  simulation  workstation 
system  (see  architecture  in  Figure  5)  that  required  the  operator  to  manage  several  TLAM  missions 
(already  in  progress)  simultaneously.  The  TSC  monitors  and  adjusts  the  TLAM  strikes  as  events 
unfold.  TSC  tasks  include  re-targeting  missiles,  assessing  missile  coverage  zones  and  remembering 
which  missiles  are  striking  which  targets,  and  responding  to  questions  presented  through  a  chat 
interface.  The  participants  were  LMATL  personnel,  not  U.S.  Navy  TLAM  experts. 

Cognitive  Bottleneck — Working  Memory 

Working  memory  is  that  part  of  the  human  cognitive  system  that  one  is  consciously  aware 
of  at  any  time.  Working  memory  can  be  thought  of  as  analogous  to  RAM  memory  on  a  computer. 
While  memory  storage  capacity  in  the  human  cognitive  system  is  vast,  the  majority  is  permanent 
memory  (analogous  to  hard-disk  storage  on  a  computer).  Meaningful  communication  requires  input 
of  a  unit  of  speech  or  text  into  working  memory  to  interpret  it.  Cognitive  research  indicates  that 
working  memory  contains  two  unique  components:  verbal  and  spatial.  Successfully  augmenting 
a  Working  Memory  bottleneck  requires  gauges  that  measure  the  spatial  and/or  verbal  working 
memory. 

The  LMATL  research  focused  on  measuring  verbal  and  spatial  working  memory  through  a  neural 
network  that  produced  two  gauge  values  based  on  a  mix  of  pre-processed  data  and  ICA  gauge  output. 
The  gauges  were  based  on  the  Pupillometry  Index  of  Cognitive  Activity  (ICA)  (Eye  Tracking,  Inc.), 
physiological  sensors  (GSR,  ECG)  that  provided  general  workload,  and  EEG  that  provided  localiza¬ 
tion  of  verbal/spatial  working  memory.  Gauge  values  were  then  applied  to  a  rule-based  system  that 
determined  when  the  augmentation  should  be  turned  on  or  off. 

Mitigation 

Intelligent  sequencing  improved  TSC  performance.  The  system  received  updates  regarding  the 
operator’s  current  use  of  spatial  and  verbal  working  memory  through  the  Artificial  Neural  Network 
(ANN).  When  the  new  task  was  proposed,  the  system  decided  to  either  present  the  task  immediately 
or  at  a  later  time  when  the  operator’s  working  memory  was  less  tasked.  When  the  system  recognized 
that  the  operator  was  simultaneously  engaged  in  the  re-targeting  task  and  the  message  alert  task,  and 
was  experiencing  a  verbal  working  memory  overload,  the  alerts  would  be  deferred  until  the  location 
task  (highly  spatial)  was  presented,  and  then  the  verbal  alerts  were  delayed. 

Independent  and  Dependent  Variables 

The  experiment  used  a  single-factor  repeated  measures  experimental  design.  The  independent 
variable  was  Mitigation  ON  or  Mitigation  OFF.  The  dependent  variable  was  the  performance  score 
for  the  TLAM  re-targeting  task  (weighted  by  complexity).  Emergent  target  tracks  appear  on  the 
TTWCS  display,  the  TSC  must  process  weapon  profile  (time  of  flight)  of  TLAMs  already  in  flight 
to  previously  assigned  targets,  and  the  number  of  TLAMs  required  for  the  new  emergent  target  (1,2, 
or  3).  Based  on  those  factors,  the  operator  must  decide  which  in-flight  TLAMs  to  re-target  to  what 
higher  priority  emergent  targets. 

2.3.5  Results 

The  DARPA  goal  was  to  achieve  a  500%  improvement  in  the  Working  Memory  bottleneck. 

The  Working  Memory  (verbal)  bottleneck  was  measured  by  number  of  successfully  serviced 
emergent  targets  weighted  by  target  complexity.  The  results  obtained  were  statistically  significant  at 
the  p  <  0.001  level,  and  were  done  within  system  performance  constraints  specified  at  the  start 
of  IWIIUS  Phase  II.  As  can  be  seen  in  Table  9,  this  goal  was  surpassed  for  the  bottleneck  addressed 
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by  the  LMATL  team.  In  the  Working  Memory  bottleneck  (measured  by  the  retargeting  task),  a  642% 
improvement  was  observed,  meaning  that  the  TSC’s  ability  to  re-target  TLAMs  in  flight  to  new 
emergent  targets  was  increased  over  sixfold.  This  improvement  occurred  in  the  presence  of  compet¬ 
ing  tasks  to  answer  alerts  (chat  questions)  that  tested  the  participant’s  situational  awareness  and 
ability  to  assess  coverage  zones  in  the  location  task,  which  has  the  potential  to  allow  a  TSC  to  do  all 
tasks  related  to  execution  of  a  multiple  TLAM  launch  that  currently  is  accomplished  in  several  layers 
by  three  different  operators. 

2.3.6  Phase  III  Goals 

Phase  III  will  provide  more  challenges  as  IWIIUS  technology  is  applied  to  increasingly  mature 
TTWCS  test  beds  in  more  operationally  defined  scenarios  with  refined  task  representation.  Some 
other  challenges  include  the  following: 

•  Under  operational  conditions  of  cognitive,  physiological,  and  environmental  stress,  mitigate 
the  impact  of  platform-specific  stressors  on  cognition  and  information  processing  performance 

•  Moving  as  many  biosensors  off  the  head  as  possible — eye  pupillometry  and  tracking  off  the  head 

•  Inserting  remaining  sensors  in  a  single  integrated  helmet  design 

•  Integrating  the  fNIR/EEG  sensors 

•  Developing  truly  task-independent  gauges 

•  Developing  additional  mitigation  strategies  for  other  bottlenecks 

•  Improving  Cognitive  State  Assessor  “fieldability”  and  flexibility 

•  Evaluating  aspects  of  task  interference 

•  Developing  an  error-prediction  and  early  warning  gauge 

Finally,  system  degradation  will  be  addressed.  In  its  current  configuration,  system  performance 
will  degrade  significantly  with  the  loss  of  one  sensor.  This  issue  will  be  addressed  in  the  future  by 
training  the  neural  network  with  a  more  robust  training  set,  including  situations  with  inactive  sensors. 
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2.4  AIR  FORCE 


2.4.1  U.S.  Air  Force — Unmanned  Air  Vehicle  (UAV)  Operator  for  the  Joint  Unmanned  Combat 
Air  System  (J-UCAS)  Concept  Validation  Experiment  (CVE)  Analysis  Report  Summary 


Table  10.  Boeing®  Executive  Summary  of  approach  and  results. 


Team:  Boeing® 

Application:  Unmanned  Air  Vehicle  (UAV)  Operator  (combat) 

Hard  Problems: 

•  UAVs  currently  require  multiple  operators  to  control  single  UAV.  J-UCAS  requires  single 
operator  and  multiple  vehicles 

•  Controlling  multiple  UAVs  is  cognitively  demanding  because  of  constant  context  shifting 
within  and  between  tasks  for  each  vehicle  and  between  vehicles 

Transition: 

•  Joint  -  Unmanned  Combat  Air  System  (J-UCAS) 

•  POC:  Mark  Pitarys  -  Deputy  Program  Manager  J-UCAS  (DARPA) 

Test  Conditions: 

Scenario 

•  UAV  Operator  workstation — managing  122  vehicles 

•  Managing  multiple  vehicles  and  keeping  track  of  multiple  tasks  and  procedures  for 
each  vehicle  under  time  constraints,  creating  high  demands  on  Executive  Function 
(cognitive  processing  capacity) 

•  Complex  task  environment  with  competing  tasks: 

o  Task:  Tactical  Situation  Display  (TSD) — Process  multiple  vehicles — target 
identification,  weapons  pairing,  and  attack 
o  Task:  Vehicle  Health  Task  (VHT) — Process  pop-up  failures  and  address 

maintenance/logistic  issues 

Participants 

•  N  =  3 

•  Boeing  &  USAF  Engineers:  Highly  trained,  right-handed  males 

Gauges 

Gauge  &  Developer: 

Tested: 

Used: 

NuWAM  TSD  part-task  (AFRL) 

y 

✓ 

NuWAM  VHT  part-task  (AFRL) 

v 

SPC  TSD  part-task  (Boeing®) 

v 

✓ 

SPC  VHT  part-task  (Boeing®) 

✓ 

✓ 

NovaSol:  fNIR,  tested  and  used  (SPC  and  NuWAM) 

✓ 

ETI:  Eyetracking/Pupillometry  (ICA),  tested  and  used 
(SPC  and  NuWAM) 

v 

✓ 

AFRL:  EEG,  ECG,  EOG,  NN,  tested  and  used 
(NuWAM) 

v 

✓ 

QUASAR  -  dry  electrodes,  tested  and  used  (NuWAM) 

V 

✓ 

Clemson:  Arousal  Meter,  tested  and  used  (NuWAM) 

V 

✓ 

2  J-UCAS  target  is  one  operator  controlling  four  air  vehicles.  The  AugCog  goal  is  one  operator  doing  the  work  of 
three.  Thus  the  scenario  design  was  one  operator  operating  12  vehicles. 
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Table  10.  Boeing®  Executive  Summary  of  approach  and  results,  (cont) 


Integration  Architecture 

•  Sensor  data  collected/distributed  via  networked  database 

•  Cognitive  State  Assessor  (CSA)  inputs  sensor  data  from  database 

•  CSA  computes  TSD  and  VHT  gauge  values  back  to  database 

o  Indicates  high  or  low  task  derived  (TSD  or  VHT)  cognitive  processing  load 

•  Augmentation  Manager  combines  gauge  data  and  context  data  to  determine 
mitigation  triggers  based  on  rules 

o  Context  is  determined  by  operator  behaviors  and  number  of  impending 
critical  tasks 

■  Context  gauges  were  developed  to  complement  each  cognitive 
gauge  based  on  number  and  type  of  operator  inputs  expected  by  the 
system  for  pending  tasks 

o  The  mitigation  is  turned  on  when  the  cognitive  gauge  and  the  context  gauge 
agree 

3 

o  A  predefined  set  of  mitigations  is  assigned  to  a  single  gauge 
o  Mitigation  initiation  and  termination;  a  joint  function  of  gauge  output  and 
computed  context 

Mitigation  Rationale 

•  During  CVE,  a  predefined  set  of  mitigations  was  triggered  by  the  AM  (dependent 
on  cognitive  state  and  context  gauges) 

Experimental  Design 

•  Competing  tasks  paradigm 

•  Within  subject,  2x1  repeated  measure 

Independent  Variables 

•  Augmented  versus  non-augmented 

Dependent  Variables 

•  Executive  Function:  TSD — number  of  bombs  on  target 

•  Working  memory:  VHT — number  of  tasks  missed 

•  Sensory  input:  number  of  events  performed  simultaneously 

•  Attention:  number  of  detected  and  engaged  pop-up  events 

Bottleneck 

Bottleneck 

Goal 

Sample 

Tested 

Team  Results 
(by  Task) 

*Mitigation 

Practical 

Demo 

Statistically 

Significant4 

Working 

Memory 

500% 

(«  =  3). 

680% 

Mitigation  1 

✓ 

W(6)  = -21,  p<  0.025 

Executive 

Function 

100% 

(n  =  3). 

241% 

Mitigations  2, 

3,  5,  6 

✓ 

W(6)  =-19,  p<0.05 

Sensory  Input 

100% 

(n  =  3). 

283% 

Mitigations  1, 

2,  3,  4,  5,  6,  7 
and  8 

✓ 

W(6)  = -21,  p<  0.025 

Attention 

100% 

(«  =  3). 

750% 

Mitigations  2, 
3,4.  5,  6 

✓ 

W(6)  =  -21,  p<  0.025 

*See  Table  1 1  on  next  page  for  mitigation  guide 


3  Each  gauge  can  have  multiple  mitigations. 

4  (Wilcoxen  -  signed  pair) 
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Table  11.  Mitigation  guide. 


Task 

Mitigation 

Name 

Bottleneck(s)  Mitigated 

Mitigation  Explanation 

1-VHT 

Alert  Manager 

Working  Memory 

The  Alert  Manager  is  a  dialog  box  that  collects  all 
of  the  currently  active  vehicle  alerts  into  a  single 
list  sorted  by  priority. 

2-TSD 

De-clutter 

Executive  Function, 

Sensory  Input,  Attention 

The  de-clutter  mitigation  places  a  “fog  layer”  over 
the  map,  thereby  partially  obscuring  the  map  and 
the  vehicle  flight  path  lines.  This  makes  the 
details  of  interest  (e.g.,  air  vehicles  and  targets) 
much  easier  to  see. 

3-TSD 

Earcons 

Executive  Function, 

Sensory  Input,  Attention 

Earcons  are  brief,  distinctive  sounds  that  are 
typically  associated  with  specific  mission  events. 
For  example,  a  camera  shutter  “click”  sound  is 
presented  to  signal  initiation  of  a  Capture  SAR 
command.  Auditory  cue  consistent  with  event 
initiation  (redundant  to  menu  selection  -  intent  to 
reduce  error  rate  with  auditory  task  initiation  cue). 

4-TSD 

Time  Critical 
Target  (TCT) 
Voice 

Sensory  Input,  Attention 

The  time-critical  voice  mitigation  is  a  computer¬ 
generated  female  voice  that  speaks  the  words 
“Tiger  leader,  we’ve  detected  a  time  critical 
target”  whenever  a  time-critical  target  is 
activated.  Auditory  alert  message  is  specific  to 
TCT. 

5-TSD 

Bookmark 

Executive  Function, 

Sensory  Input,  Attention 

The  bookmark  mitigation  provides  the  operator 
with  some  assistance  in  determining  the  most 
expedient  “next”  action  on  the  Tactical  Situation 
Display.  Each  strike  package  has  an  icon 
traveling  with  it  at  the  center  of  mass  of  the  four 
air  vehicles.  The  circular  center  section  of  the 
icon  is  color-coded  red,  yellow,  or  green, 
depending  on  the  relative  urgency  of  the  most 
urgent  vehicle  for  each  of  the  strike  packages — 
visual  cue  for  recommending  next  UAV  operator 
tasks. 

6-TSD 

Unified 

Bookmark 

Executive  Function, 

Sensory  Input,  Attention 

The  unified  bookmark  expands  on  the  basic 
bookmark  by  including  in  its  logic  the  additional 
considerations  of  Vehicle  Health  Task  alerts.  A 
red  square  is  placed  on  the  next  recommended 
action-visual  cue  for  next  UAV  operator  task. 

7-TSD 

UAV  Menu 
Tokens 

Sensory  Input 

The  UAV  Menu  Tokens  mitigation  placed  two 
“tokens”  on  each  air  vehicle  icon.  The  tokens 
were  small  squares.  One  token  was  red  and  one 
was  green.  The  red  token  corresponded  to  the 
Direct  Attack  option  on  the  drop  down  menu  and 
the  green  token  corresponded  to  the  SAR/Attack 
option-menu  shortcut. 

8-VHT 

Vehicle  Health 
Tones 

Sensory  Input 

The  Vehicle  Health  Tones  mitigation  presented  a 
unique  computer-generated  tone  for  each  strike 
package  whenever  a  vehicle  health  problem  was 
detected — auditory  cue  to  a  pending  VHT  and 
localization  (which  UAV  SP). 
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2.4.2  Background 

The  Boeing8  team  analysis  report  discusses  the  Cognitive  Task  Environment(s),  human  state 
gauges  and  biosensors  used,  research  methods,  integrated  prototype  system  design,  and  empirical 
results.  Boeing8  Phantom  Works  was  the  chosen  performer  to  work  on  IWIIUS  in  an  Unmanned 
Air  Vehicle  (UAV)  domain.  Table  10  is  a  summary  of  the  approach  and  results  of  manipulating  the 
cognitive  state  of  a  Joint  Unmanned  Combat  Air  System  (J-UCAS)  UAV  operator.  The  primary 
bottleneck  assigned  to  Boeing8  was  the  Executive  Function  bottleneck;  however,  the  team  also 
researched  Working  Memory,  Sensory  Input,  and  Attention.  The  Boeing8  research  environment  was 
very  complex,  with  biosensors  fed  to  an  ANN  for  Executive  Function  and  Attention,  and  separately 
through  another  process  called  Statistical  Process  Control  (SPC)  for  Working  Memory  and  Sensory 
Input.  For  the  Executive  Function  bottleneck,  three  biosensors  were  used:  EEG,  ECG,  and 
pupilometry  ICA. 

Development  and  Analysis  Team 

The  Boeing8  team  was  led  by  Boeing8  and  included  Air  Force  Research  Laboratory  (AFRL), 
NOVASOL,  University  of  Clemson,  and  Eyetracking,  Inc.  For  Phase  II,  the  team  developed 
the  simulated  J-UCAS  workstation  within  a  CLIP  based  on  operational  environments  expected 
of  the  U.S.  Air  Force’s  Umnanned  Combat  Air  System. 

2.4.3  Operational  Application — Unmanned  Air  Vehicle  (UAV)  Operator 

The  J-UCAS  program  is  a  joint  DARPA/Air  Force/Navy  effort  to  demonstrate  the  technical 
feasibility,  military  utility,  and  operational  value  for  a  networked  system  of  high-performance, 
“weaponized”  UAVs  to  effectively  and  affordably  prosecute  21st  century  combat  missions,  including 
SEAD,  surveillance,  and  precision  strike  within  the  emerging  global  command  and  control  architec¬ 
ture. 

The  initial  operational  role  for  the  J-UCAS  desired  for  the  Air  Force  is  as  a  “first-day-of-the-war” 
force  enabler  that  will  complement  a  strike  package  by  performing  the  Suppression  of  Enemy  Air 
Defense  (SEAD)  mission  through  lethal  and  non-lethal  means.  In  this  role,  J-UCAS  would  accom¬ 
plish  pre-emptive  destruction  and  electronic  suppression  of  sophisticated  enemy  integrated  air 
defense  systems  (IADS)  in  support  of  manned  strike  packages.  Throughout  the  rest  of  the  campaign, 
J-UCAS  would  provide  continuous  vigilance  with  an  immediate  lethal  strike  capability  to  prosecute 
high  value  and  time-critical  targets.  After  the  conflict,  the  J-UCAS  could  fly  peacekeeping  missions, 
such  as  enforcing  “no-fly”  zones;  these  missions  typically  entail  flying  long  hours  of  patrols  (so- 
called  “dull”  missions). 

The  initial  operational  role  for  the  Navy’s  J-UCAS  is  to  provide  carrier-based,  survivable,  and 
persistent  surveillance,  reconnaissance,  and  targeting  to  complement  manned  assets  and  long-range 
precision  strike  weapons.  But  to  fully  exploit  its  potential  and  “buy  its  way”  onto  the  carrier,  SEAD 
and  Strike  capabilities  will  be  incorporated  into  the  design  from  the  outset  and  fully  developed 
in  future  spirals.  The  system  will  be  seamlessly  integrated  with  manned  aircraft  missions,  carrier 
air  traffic  control,  and  deck  operations,  as  well  as  with  the  carrier’s  Command,  Control,  Communi¬ 
cations,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR)  architecture. 
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“Hard  Problems”  for  the  UAV  Operator 

UAVs  currently  require  multiple  operators  to  control  a  single  UAV.  J-UCAS  requirements 
dictate  a  single  operator  controlling  multiple  vehicles.  The  operator  must  manage  multiple  “flights” 
(containing  up  to  four  UAVs  in  each  flight)  of  vehicles.  The  tasks  include  target  identification, 
weapons  pairing,  and  attack.  During  this  already  complex  task,  the  operator  must  also  deal 
with  vehicle  system  failures  and  logistic  issues.  All  operations  are  conducted  under  critical 
time  constraints  that  create  high  demands  on  cognitive  processing  capacity  (Executive  Function). 
Controlling  multiple  UAVs  is  cognitively  demanding  because  of  constant  context  shifting  within 
and  between  tasks,  which  places  a  heavy  demand  for  executive  function  and  for  each  vehicle  and 
between  vehicles,  a  divided  attention  cognitive  demand. 

Transition  Customer 

The  Boeing8  team  is  working  primarily  with  the  U.S.  Air  Force  as  they  are  the  initial  transition 
customer  for  J-UCAS.  The  U.S.  Navy  is  also  interested  in  a  carrier-capable  version  of  the  J-UCAS. 
The  J-UCAS  office,  under  DARPA  leadership,  with  support  from  the  services,  is  focused  on  plan¬ 
ning  and  executing  a  demonstration  program  that  supports  Air  Force  and  Navy  future  requirements. 
Operation  Assessments  (OAs)  will  follow  the  Demonstration  program  beginning  in  FY  2007.  The 
OAs  are  expected  to  provide  the  services  with  several  program  options  in  FY  2007-2009.  A  Joint 
Requirements  Group,  composed  of  the  Joint  Staff,  Air  Force,  and  Navy,  will  coordinate  with  the 
Joint  Forces  Command  and  other  combatant  commanders  to  develop  and  validate  J-UCAS 
requirements. 

Test  Conditions 

The  Boeing8  prototype  system  is  designed  around  the  CSA.  The  CSA  inputs  biosensor  data 
collected  from  a  networked  database  and  computes  the  Tactical  Situation  Display  (TSD)  and  Vehicle 
Health  Task  (VHT)  gauge  values  (cognitive  load).  The  Augmentation  Manager  combines  gauge  data 
and  context  data  to  determine  mitigation  triggers  based  on  a  rule  set.  Mitigation  initiation  and  termi¬ 
nation  is  a  joint  function  of  gauge  output  and  context  in  an  “and”  statement — they  must  agree.  The 
computed  CSA  gauge  values  came  from  two  different  approaches  during  research  and  development. 
One  approach  was  based  on  an  ANN  that  received  EEG,  ECG,  EOG,  pupilometry,  and  fNIR  sensor 
data.  The  ANN  was  used  to  detect  the  Attention  and  Executive  Function  bottlenecks.  The  other 
approach  was  based  on  a  Statistical  Process  Control  (SPC)  model  that  received  fNIR  and  pupillo- 
metry  sensor  data.  The  SPC  model  was  used  to  detect  the  Sensory  and  Working  Memory  bottlenecks. 
The  Integration  architecture  used  for  the  Boeing8  CVE  is  shown  in  Figure  6  on  the  next  page. 
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Physiological  activity -  |  User  Interface 


Figure  6.  Boeing®1  integration  architecture. 


2.4.4  Methodology 

The  test  conditions  for  the  Boeing®  team  consisted  of  a  UAV  operator  console  that  housed  two 
19-inch  flat-panel  displays.  The  left  display  presented  the  TSD  while  the  right  displayed  the  targeting 
(Synthetic  Aperture  Radar  [SAR]  image  and  Designated  Mean  Point  of  Impact  [DMPI]  selection 
tool)  and  VHTs.  For  the  Executive  Function  bottleneck,  the  operator,  after  finishing  a  VHT  or 
performing  a  targeting  task  with  the  SAR  image,  had  to  switch  tasks  to  the  TSD.  Once  switching 
back  to  the  TSD,  the  operator  had  to  correctly  pair  the  vehicle  to  target  (12  vehicles  to  choose  from) 
with  the  least  amount  of  time  remaining  before  overflight  (or  would  miss  the  target). 

Cognitive  Bottleneck — Executive  Function 

The  Executive  Function  (sometimes  called  the  Supervisory  Function)  monitors  and  controls 
ongoing  mental  operations  and  actions.  It  selectively  activates  some  memories  and  processes  and 
inhibits  irrelevant  ones.  Executive  Function  is  critical  to  tactical  decision-making  that  requires  split- 
second  assessment  of  the  tactical  situation.  The  assessment  of  the  tactical  situation  can  be  easily  lost, 
and  the  operator  must  be  able  to  recover  context  quickly. 

Mitigation 

The  Boeing®  team  based  their  mitigation  strategies  on  cue  memory  retrieval  to  improve  perform¬ 
ance  when  Executive  Function  was  overloaded.  The  goal  was  to  maximize  executive  functioning 
and  facilitate  memory  enhancement.  The  system  used  four  mitigation  techniques:  Map  De-clutter, 
Earcons,  Book  Marking,  and  Unified  Book  Marking.  The  details  of  these  techniques  are  listed 
in  Table  1 1  and  in  the  Boeing®  analysis  report. 

Cognitive  Bottlenecks — Working  Memory,  Sensory  Input,  Attention 

The  Boeing®  team  also  successfully  researched  mitigation  techniques  for  the  remaining  three 
bottlenecks:  Working  Memory,  Sensory  Input,  and  Attention.  The  test  conditions  were  the  same 
as  for  the  Executive  Function  bottleneck,  with  different  task  metrics. 

Independent  and  Dependent  Variables 

The  experiment  used  a  competing  tasks  paradigm,  within  subject,  2x1  repeated  measures. 

The  independent  variable  was  Mitigation  ON  or  Mitigation  OFF.  The  dependent  variables  were 
number  of  bombs  on  target  (Executive  Function),  number  of  vehicle  health  tasks  missed  (Working 
Memory),  number  of  events  performed  simultaneously  (Sensory  Input),  and  number  of  detected 
and  engaged  pop-up  events,  with  versus  without  augmentation. 

2.4.5  Results 

The  DARPA  goal  was  to  achieve  a  100%  improvement  in  the  cognitive  Executive  Function 
bottleneck.  The  results  obtained  were  statistically  significant  at  the  p<0.5  level,  and  were  achieved 
within  system  performance  constraints  specified  at  the  start  of  IWIIUS  Phase  II.  As  can  be  seen  in 
Table  10,  this  goal  was  matched  or  surpassed  for  those  bottlenecks  addressed  by  the  Boeing®  team. 
For  the  Executive  Function  bottleneck  (measured  by  number  of  successful  bombs  on  target),  a  241% 
improvement  was  observed,  meaning  that  the  operators’  ability  to  strike  assigned  targets 
was  increased  nearly  two  and  a  half  times  when  the  mitigation  techniques  were  implemented. 
Improvement  results  for  Working  Memory,  Sensory  Input,  and  Attention  bottlenecks  were  680%, 
283%,  and  750%,  respectively. 
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2.4.6  Phase  III  Goals 

Phase  III  will  provide  more  challenges  as  IWIIUS  technology  is  applied  to  increasingly  refined 
J-UCAS  operator  environments.  Challenges  include  the  following: 

•  Under  operational  conditions  of  cognitive,  physiological,  and  environmental  stress,  mitigate 
the  impact  of  platform-specific  stressors  on  cognition  and  information  processing  performance 

•  Developing  multi-level  gauges  and  augmentation  management  to  increase  system  stability 
with  high-performance  mitigations 

•  Making  the  cognitive  gauges  and  Augmentation  Manager  proactive  rather  than  reactive 

•  Developing  and  testing  gauges  that  are  highly  correlated  with  specific  cognitive  bottlenecks 

•  Incorporating  context  modeling  and  workload  avoidance  strategies 

•  Decreasing  time  for  ANN  training  (currently  takes  too  long  and  must  be  performed  too  often, 
perhaps  every  few  hours) 

•  Increasing  the  redundancy  of  the  sensors  and  gauges  to  increase  the  graceful  degradation 
capability  of  the  system 

•  Investigating  wireless  technology  to  increase  the  mobility  of  the  operator 

•  Improving  sensor  fieldability  (competition  for  space  on  the  operator,  ergonomics,  ruggedization, 
etc.) 

•  Increasing  operational  fidelity  of  the  system  by  expanding  existing  scenarios  to  include  wider 
range  of  tactical  problems  and  adding  mission  control  station  functionality 

•  Must  clearly  demonstrate  3-to-l  manpower  reduction 

A  key  area  of  research  for  Phase  III  and  technology  transition  for  Phase  II  and  technology 
transition  that  warrants  further  investigation  is  architecting  gauges  that  are  an  integration  of  multiple 
sensors  (using  ANN,  SPC,  or  both).  Gauges  designed  with  this  philosophy  are  expected  to  be  more 
robust,  and  with  a  more  graceful  degradation  response  to  failures  than  single-sensor  systems. 

Additionally,  the  development  of  an  AugCog  system  could  include  methods  for  using  part-tasks 
to  baseline  gauge  readings  for  a  given  “task  type.”  That  knowledge,  along  with  knowledge  of  the 
operator’s  current  cognitive  state,  could  be  used  to  predict  the  occurrence  of  cognitive  overload  when 
starting  the  next  task.  This  information  could  be  used  for  a  proactive  workload  avoidance  approach 
in  the  design  of  the  augmentation  manager. 
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3.  DISCUSSION 


Humans  are  constrained  in  the  amount  of  information  they  can  manage  at  any  given  moment 
(Jiang  and  Kanwisher,  2003;  Allport,  1993).  DARPA,  recognizing  the  operational  implications  of 
this  problem  for  the  military,  has  sponsored  the  Improving  Warfighter  Information  Intake  under 
Stress  (IWIIUS)  program.  This  program  was  developed  to  specially  address  methods  for  detecting 
and  mitigating  limitations  of  human  information  processing,  and  designing  solutions  to  enhance  the 
exchange  and  use  of  information  in  man-machine  systems.  This  report  is  a  compilation  of  the  results 
of  the  second  year,  or  Phase  II,  of  the  program.  During  this  phase  of  the  program,  teams  demonstrat¬ 
ed  how  the  effectiveness  of  warfighters  can  be  improved  by  making  the  human-computer  interface 
respond  to  specific  human  performance  capabilities  and  limitations  in  stressful  operational 
environments. 

Phase  II  extended  previous  work  by  focusing  on  the  development  of  closed-loop  prototype  systems 
that  addressed  theoretically  derived  cognitive  bottlenecks.  The  closed-loop  integrated  prototypes 
(CLIPs),  developed  by  each  industry  team,  rely  on  computational  systems  to  determine  the  state 
of  the  cognitive  bottleneck  in  a  warfighter  in  real  time.  Whenever  the  warfighter  approached  cogni¬ 
tive  limits  (minimum  or  maximum)  of  one  or  more  of  the  cognitive  bottlenecks,  mitigation  strategies 
were  implemented,  given  the  warfighter’s  current  task  requirements,  to  reallocate  resources  or  redi¬ 
rect  efforts  of  the  warfighter  to  enhance  overall  performance.  Optimizing  the  warfighter’s  cognitive 
state  through  these  mitigation  strategies  reduced  the  limiting  affects  of  the  cognitive  bottlenecks, 
thus  maximizing  information  throughput  of  the  warfighter.  Each  of  the  four  teams  investigated 
a  different  operationally  relevant  task  specifically  addressing  one  or  more  of  the  cognitive  processing 
bottlenecks  identified  by  the  DARPA  management  team.  The  management  team  also  established 
performance  goals  for  each  of  the  cognitive  processing  bottlenecks  as  criteria  for  success  during 
Phase  II. 

With  these  goals  in  mind,  the  efforts  of  each  team  sought  to  demonstrate  the  possibility  that  closed- 
looped  systems  could  be  used  to  (1)  enhance  operational  effectiveness  through  specific  improve¬ 
ments  to  warfighter  efficiency,  (2)  increase  the  amount  of  information  operators  can  handle, 

(3)  reduce  manpower  requirements  (e.g.,  one  person  doing  job  of  two  or  more),  and  (4)  improve 
attention  management  during  stressful  operations.  Each  industry  team  provided  detailed  results  (see 
Volume  II).  Their  accomplishments  are  a  significant  step  forward  in  the  application  of  augmented 
systems  within  each  of  the  operational  environments  investigated. 

The  Honeywell"  team  exceeded  the  objectives  of  Phase  II  by  demonstrating  that  a  closed-loop 
computational  system  can  improve  performance  in  a  simulated  dismounted-military  environment 
(infantry).  They  showed  that  when  appropriate  mitigations  were  applied  at  appropriate  times,  signi¬ 
ficant  improvement  in  attention  could  be  obtained.  Based  on  post-hoc  analyses  and  comparison 
with  critical  events  embedded  in  the  test  scenarios  where  high  attentional  demands  were  expected, 
the  Attention  bottleneck  gauges  appeared  to  accurately  reflect  high  attention  demands  98%  of  the 
time.  The  Honeywell '  cognitive  gauges  could  detect  state  changes  within  250  to  300  ms  of  the  onset 
of  the  critical  events,  and  mitigations  were  invoked  within  1  second  of  the  critical  event.  These 
improvements  are  well  within  the  system  performance  goals  established  for  Phase  II. 

The  success  of  the  DCC  team  serves  as  the  benchmark  for  performance  improvement  for  military 
vehicle  operators.  Their  results  show  a  significant  performance  improvement  on  competing  tasks 
when  the  closed-loop  augmentation  system  was  activated.  Without  the  intervention,  competing  task 
performance  was  significantly  lower  during  periods  of  high  workload.  The  empirical  results  demon¬ 
strate  improvements  of  108%  for  the  sensory-auditory  bottleneck,  exceeding  the  program  goal 
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for  the  sensory  bottleneck.  A  72%  improvement  for  the  visual-sensory  bottleneck  was  also  achieved. 
This  result  is  a  significant  improvement,  especially  considering  that  they  were  using  real-world 
driving  tasks  on  public  roads.  These  tasks  are  highly  demanding  of  sensory  systems,  leaving  few 
resources  available  for  competing  visual  tasks.  A  103%  improvement  for  working  memory  was 
shown  over  the  baseline  (non-mitigated)  condition.  Their  results  demonstrate  that  it  is  possible  to 
detect  cognitive  states  changes  in  real  time  and  that  this  information  can  be  used  to  provide  a 
significant  improvement  in  operator  performance  under  closed-loop  conditions  in  the  real-world 
environment. 

The  long-range  implications  of  the  DCC  augmented  cognition  technology  in  the  LAV  environment 
is  twofold:  First,  the  workload  of  the  crew  could  be  dynamically  allocated  between  the  driver,  vehicle 
commander,  and  additional  vehicle  occupants  by  (1)  better  sharing  of  various  tasks  based  upon 
the  cognitive  state  of  each  crew  member  and  the  autonomous  capabilities  of  the  vehicle,  (2)  by  using 
the  augmented  cognition  system  to  focus  the  crew’s  attention  on  the  most  critical  task,  and 
(3)  by  having  the  augmented  cognition  system  report  on  the  cognitive  status  of  crewmembers 
(during  over-  and  under-workload  conditions).  Second,  the  driver’s  ability  to  safely  operate 
the  vehicle  would  be  enhanced  by  (1)  automating  and  assisting  in  driver  functions  when  appropriate, 
and  (2)  prioritizing  caution  and  warning  indicators  through  the  most  effective  sensory  channel.  These 
applications  will  be  further  explored  in  Phase  III. 

LMATL’s  neural  network  successfully  classified  working  memory  in  the  overloaded  or  under¬ 
loaded  state  over  90%  of  the  time.  The  workload  classifications  took  place  in  under  500  ms,  and 
the  mitigation  was  triggered  5  seconds  from  the  time  a  working  memory  exceeded  threshold. 

The  results  show  a  642%  improvement  in  working  memory  as  measured  by  successful  re-targeting 
performance  when  the  intelligent  sequencing  mitigation  was  used.  These  results  are  operationally 
relevant  to  future  TTWCS  development  since  the  added  dimension  of  re-targetable  Tomahawk 
missiles,  in  addition  to  the  pressure  for  manning  reduction,  will  dramatically  increase  the  frequency 
with  which  TTWCS  operators  will  experience  working  memory  overload.  LMATL  identified  several 
problems  with  current  sensor  systems,  and  plan  to  investigate  the  feasibility  of  using  non-contact 
sensors  during  Phase  III  for  an  extended  period  in  an  operational  environment.  The  LMATL’s  CLIP 
met  all  requirements  for  successfully  addressing  the  Working  Memory  bottleneck. 

The  Boeing®  team  conducted  a  series  of  experiments,  refining  the  bottleneck  gauges  and  mitiga¬ 
tion  strategies  iteratively  based  on  empirical  results  throughout  the  development  process.  A  subset 
of  data  obtained  from  these  studies  was  used  in  assessing  the  bottleneck  results.  The  Boeing®  team 
met  the  performance  goals  for  all  the  cognitive  bottlenecks,  demonstrating  a  241%  improvement 
for  their  assigned  bottleneck  of  Executive  Function.  Cognitive  state  detections  were  successfully 
categorized  into  their  appropriate  state  92%  of  the  time,  within  1  second  of  its  changing.  Mitigations 
were  implemented  within  1  second  of  it  being  requested  by  the  augmentation  manager.  These  results 
exceeded  the  program  goals  of  achieving  a  100%  improvement  in  executive  function,  with  90%  or 
better  accuracy  in  less  than  2  seconds  after  a  state  change,  and  mitigating  in  less  than  1  minute. 

The  performance  differences  observed  between  the  augmented  and  non-augmented  conditions 
were  statistically  and  operationally  meaningful. 

In  addition  to  the  general  findings  that  operator  performance  could  be  improved  using  a  variety 
of  mitigation  strategies,  multiple  lessons  were  learned,  or  additional  problems  highlighted,  that  have 
significant  implications  for  the  future  of  augmented  cognition. 
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Lessons  Learned.  The  combined  efforts  of  each  research  team  has  addressed  several  important 
issues  that  needed  to  be  identified,  elaborated  on,  or  resolved  for  the  future  development  of  opera¬ 
tional  AugCog  systems. 

The  Need  to  Understand  User.  One  fact  that  became  relatively  apparent  is  the  need  to  conduct 
a  Cognitive  Task  Analysis  (CTA)  to  fully  understand  the  decision-making  required  in  the  perform¬ 
ance  of  operational  tasks.  Having  a  clear  understanding  of  who  the  user  is  as  well  as  thoroughly 
understanding  what  it  is  about  their  job  that  is  cognitively  difficult,  is  crucial  to  developing  meaning¬ 
ful  and  effective  AugCog  systems.  There  appeared  to  be  a  direct  correlation  between  the  quality  and 
depth  of  the  cognitive  task  analyses  conducted  at  the  start  of  Phase  II,  the  ease  with  which  the  CLIP 
was  developed,  and  the  ultimate  success  the  development  teams  had  in  demonstrating  an  effective 
CLIP.  One  team  (Boeing8)  had  difficulty  defining  the  context  with  which  the  warfighter  operated 
(largely  because  the  role  of  the  warfighter  in  this  environment  is  still  ill-defined).  For  another  team 
(LMATL),  application  focus  changed  (from  Aegis  to  Tomahawk)  mid-way  through  Phase  II  develop¬ 
ment,  which  meant  that  much  of  the  thinking  about  the  hard  cognitive  problems  done  earlier  had 
limited  applicability  to  their  final  CLIP.  Several  teams  had  difficulty  distinguishing  between 
a  traditional  task  analysis,  which  focuses  more  on  what  the  operator  is  doing,  and  cognitive  task 
analysis,  which  focuses  more  on  the  decisions  made  and  thought  processes  required  to  complete 
the  task.  Current  research  efforts  had  limited  opportunity  to  investigate  specific  characteristics  that 
a  military  user  is  often  exposed  to  that  can  affect  operational  performance,  such  as  fatigue,  periodic 
changes  in  cognitive  demand,  and  conflicting  demands  for  attention,  etc. 

These  findings  support  the  notion  that  a  cognitive  task  analysis  should  be  conducted  to  fully 
understand  the  decision-making  requirements  of  the  operational  performer.  Having  a  clear  under¬ 
standing  of  who  the  user  is,  as  well  as  thoroughly  understanding  what  they  do  is  essential 
for  successfully  constructing  future  AugCog  technologies. 

Integrate  Early — Integrate  Often.  During  the  course  of  Phase  II,  addressing  the  integration 
of  the  various  sensors,  computer  hardware,  and  software  early  in  the  development  cycle  paid  off 
for  developers  that  could  do  it.  The  earlier  and  more  thoroughly  the  integration  was  considered, 
the  fewer  the  setbacks  and  more  effective  the  mitigations.  Slight  modifications  in  sensors  should 
not  be  treated  superficially,  and  require  systematic  assessment  throughout  development. 

Development  Strategy.  A  project  execution  model  that  entails  continuous  (incremental)  experi¬ 
mentation  was  the  best  approach  in  meeting  the  larger  project  objectives,  rather  than  developing 
around  less  frequent  and  more  sophisticated  integration  exercises.  However,  there  is  often  some 
cost  in  terms  of  scientific  and  experimental  rigor  in  this  approach.  Perhaps  the  best  balance  is  a  mix 
of  execution  strategies  where  a  distinction  is  made  between  developmental  experimentation  and 
formal  empirical  testing.  Teams  that  were  highly  successful  conducted  step-wise  assessments  identi¬ 
fying  the  contributions  to  each  individual  modification. 

One  Sensor  Gauge  Does  Not  Fit  All.  It  has  been  interesting  to  observe  how  the  sensors  and 
cognitive  state  gauges  developed  during  Phase  I  of  this  program  have  been  applied  over  the  course 
of  Phase  II.  Some  gauges  that  were  thought  to  be  the  most  robust  during  Phase  I  have  been  found 
to  be  less  effective  and/or  useful  when  they  were  applied  to  the  various  operational  requirements 
while  other  gauges  that  were  considered  “marginal”  during  the  Phase  I  TIE  have  provided  very 
effective  results  and  operational  utility.  Gauges  that  seemed  to  work  well  for  one  of  the  teams 
in  their  application,  sometimes  worked  poorly  for  other  teams  within  their  application.  In  other 
words,  the  selection  of  specific  gauges  for  use  within  the  operational  environment  is  highly  context 
(task)-dependent. 
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Some  performers  are  still  struggling  with  the  development  of  specific  cognitive  state  gauges  and 
some  continued  to  revise  and  develop  their  gauges — which  were  constantly  changed  to  meet  the 
demands  of  the  operational  tasks.  This  approach  directed  a  great  deal  of  effort  and  resources  for  some 
teams  away  from  the  core  goal  of  developing  meaningful  and  compelling  illustrations  of  AugCog 
into  basic  research  to  better  identify  a  sensitive  gauge.  It  may  be  that  cognitive  gauges  that  demon¬ 
strate  some  utility  in  simulated  tasks  lose  their  direct  applicability  as  tasks  become  more  cognitively 
demanding.  Teams  that  ignored  the  theoretical  constructs  of  the  gauges  developed  in  Phase  I  and 
focused  their  efforts  on  integrating  their  collection  of  gauges  through  a  neural-network  (or  equiva¬ 
lent)  algorithm,  demonstrated  enhanced  performance.  A  major  problem  with  using  this  approach 
is  that  it  is  unclear  what  specific  aspect  of  cognitive  activity  was  used  to  trigger  the  augmentation 
strategy.  In  several  cases,  it  was  unclear  whether  the  performance  enhancement  was  caused  by 
appropriately  implementing  an  intervention  strategy  to  augment  cognitive  performance  or  simply 
caused  by  using  advanced  automation  or  improved  HCI.  Clearly,  the  basic  science  of  Augmented 
Cognition  is  still  in  its  infancy  and  the  specific  application  of  cognitive  state  gauges  will  require 
significant  future  investment. 

Cognitive  Bottlenecks  May  Be  Task-Specific.  The  notion  of  isolated  cognitive  bottlenecks  has 
significant  appeal  for  a  theoretical  standpoint;  however,  it  appears  that  when  applying  these  concepts 
to  real-world  tasks,  they  are  often  overlapping  or  interacting.  It  remains  to  be  seen  if  (1)  we  have  not 
adequately  defined  what  we  mean  by  cognitive  bottlenecks  and  our  current  understanding  simply 
lacks  sufficient  specificity  to  be  useful,  or  (2)  if  the  concept  of  “cognitive  bottlenecks”  even  has 
practical  utility  in  developing  AugCog  systems.  We  may  have  to  rethink  the  whole  notion  of  cogni¬ 
tive  bottlenecks  and  define  them  operationally  in  terms  of  the  specific  demands  of  decision-making 
tasks  for  a  given  application.  In  other  words,  the  operational  definitions  and  the  theoretical  defini¬ 
tions  do  not  seem  to  directly  correspond.  It  might  be  more  useful  to  define  bottlenecks  and  gauges 
in  terms  of  the  tasks  that  are  to  be  mitigated  vice  attempting  to  address  conceptual  bottlenecks.  How¬ 
ever,  regardless  the  operational  definition  of  a  specific  bottleneck,  the  net  result  of  the  mitigation 
strategy  improved  overall  task  performance. 

Artifact  detection.  Phase  II  was  one  of  the  first  serious  attempts  to  study  what  up  to  now  have 
been  largely  laboratory  phenomena  in  an  applied  military  operational  setting.  Before  IWIIUS,  experi¬ 
mental  tasks  tended  to  be  very  simple,  highly  controlled  laboratory  experiments. 

The  objective  of  the  basic  research  accomplished  during  Phase  I  was  to  identify  underlying 
physiological  phenomena  that  correlated  to  specific  cognitive  behaviors.  Phase  I  culminated  with 
the  premise  that  there  were  now  known  physiological  phenomena  that  could  be  reliably  associated 
with  cognitive  activity  and,  therefore,  real-time  or  near-real-time  gauges  could  now  be  constructed 
to  detect  these  phenomena  in  more  complex  analogs  of  operational  tasks.  Phase  II  enhanced  these 
“gauge”  technologies  and  applied  them  to  more  operationally  meaningful  laboratory  and  field-test 
settings.  As  realism  of  the  task  has  increased,  so  has  the  need  for  advanced  processing  to  rapidly 
detect  the  physiological  phenomena  of  interest  as  well  as  remove  those  unwanted  physiological 
artifacts  from  the  raw  data  that  would  prevent  the  AugCog  systems  from  detecting  the  appropriate 
cognitive  state  of  the  operator. 

One  concern  of  artifact  detection  that  was  highlighted  in  Phase  II  and  is  identified  as  an  urgent 
concern  for  the  future  advancement  of  the  science  of  augmented  cognition  is  the  ability  to  decipher 
between  physiological  changes  related  to  cognitive  activity  vice  the  physical  requirements  of  the 
task.  This  subject  was  of  vital  concern  of  the  teams  addressing  operations  in  a  moving  vehicle 
(MEFFV)  or  for  the  dismounted  soldier  (FFW)  that  have  already  run  into  issues  with  the  impact 
of  physical  (muscular)  activity  on  the  ability  to  detect  cognitive  phenomena.  Both  development 
teams  see  a  critical  need  in  addressing  this  issue  during  the  next  phase  of  development. 
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Perceived  Stability/Trust.  Another  significant  issue  critical  for  the  advancement  of  AugCog 
is  to  not  only  accurately  identify  when  a  decline  in  cognitive  performance  occurs,  but  to  also  initiate 
an  appropriate  mitigation  to  enhance  performance  in  a  predictable  manner.  Looking  across  the 
current  development  efforts,  the  triggers  used  to  initiate  most  mitigations  are  typically  not  the  same 
trigger  used  to  turn  the  mitigations  off.  The  issue  with  stability  and  predictability  of  the  mitigations 
is  further  complicated  by  a  combination  of  physiologically  based  gauges  in  conjunction  with  context- 
based  sensors.  If  the  operator  did  not  understand  the  logic,  the  rationale  for  triggering  the  onset/offset 
of  the  mitigation,  the  potential  for  disruption  and  degraded  performance  is  significant.  The  more 
complex  the  mitigations  and  potential  triggers,  the  greater  the  problem  with  them  being  perceived 
as  predictable  and  ultimately  useful. 

Another  dimension  of  system  stability  that  relates  to  the  impact  of  degraded  sensors  is  that  each 
of  the  teams  indicated  that  while  their  systems  would  continue  to  offer  some  degree  of  augmentation 
with  the  loss  of  one  or  more  cognitive  state  sensors,  the  functionality  of  their  systems  would  change. 
If  users  are  not  aware  of  the  degraded  functionality  of  the  system,  their  expectations  of  augmentation 
would  not  be  met,  and  the  system  might  be  perceived  as  unstable.  Before  such  a  system  is  deployed, 
the  issue  of  training  will  need  to  be  developed  to  allow  users  to  operate  in  the  face  of  degraded 
augmentation  (familiarization  with  system  capability  and  limitations). 

Timing.  Many  of  the  changes  in  perceived  cognitive  load  may  be  momentary,  occurring  only 
for  a  brief  time.  The  physiological  measures  used  as  gauges  for  the  industry  teams  may  detect 
a  transient  or  sustained  change  in  cognitive  activity.  For  instance,  a  P300  event  is  a  transient  measure 
that  typically  occurs  upon  the  recognition  of  a  significant  event.  Its  magnitude  has  been  used 
as  a  measure  of  surprise  or  cognitive  workload. 

However,  the  P300  occurs  briefly  and  is  typically  useful  only  in  detecting  the  start  of  an  event. 

A  different  measure  must  be  used  to  detect  low  workload. 

It  may  be  unrealistic — or  at  least  have  little  practical  application  to  have  a  system  that  is  sensitive 
enough  to  detect  and  implement  mitigation  in  terms  of  seconds.  It  is  clear  that  in  the  operational 
environment,  momentary  changes  in  one  or  more  of  the  cognitive  bottlenecks  areas  may  occur  fairly 
rapidly  (<1  minute).  Research  needs  to  be  conducted  to  investigate  optimal  timing  for  implementing 
various  mitigation  strategies. 

Hardware  Integration.  Note  that  the  time  to  set  up  the  sensor  suite  for  all  four  teams  was  on  the 
order  of  1  hour  per  test  session/participant.  Gauges  and/or  neural  networks  required  significant  time 
and  effort  for  calibration  during  each  test  session,  taking  1 0  to  45  minutes  for  each  test  run,  and 
in  some  cases,  for  each  of  several  gauges.  Experimentation  with  reusing  calibrations  across  partici¬ 
pants,  or  even  for  the  same  participant  across  multiple  runs,  was  limited;  however,  the  results  found 
by  the  four  development  teams  suggest  that  the  underlying  phenomena  that  the  gauges  are  based 
on,  as  well  as  the  artifacts  that  must  be  removed  for  reliable  gauge  perfonnance,  will  require  that 
fielded  systems  be  calibrated  before  each  use.  Efforts  must  be  expended  to  streamline  the  calibrations 
process  and  make  it  feasible  under  field  conditions. 

The  sensors  used  during  Phase  II  are  still  physically  independent.  All  four  teams  noted  that 
a  significant  technical  objective  for  Phase  III  of  the  IWIIUS  program  must  be  to  physically  integrate 
the  bio-medical  sensors  with  the  objective  of  making  them  more  robust,  easily  donned,  and  readily 
calibrated.  Placement  of  electrodes  and  the  quality  of  their  physical  contact  with  the  user  remain 
problematic.  The  teams  are,  or  anticipate,  using  some  wireless  sensor  systems  during  Phase  III; 
however,  these  systems  will  present  their  own  integration  issues,  which  have  not  yet  been  addressed 
adequately  for  field  testing. 
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Electrical  interference  remains  a  significant  issue,  though  it  was  effectively  addressed  by  most 
teams  through  trial  and  error  throughout  development.  Phase  III  will  require  that  significant  consid¬ 
eration  be  given  to  potential  interference  sources  in  the  field,  and  then  anticipating  design  solutions 
that  avoid  the  issue  in  the  first  place.  Again,  integrated  sensors  designed  for  use  in  the  field  from  the 
outset,  vice  adapted  from  laboratory  or  medical  applications,  are  likely  to  address  this  issue  adequate¬ 
ly.  ' 

Context  Gauges.  Most  development  within  the  IWIIUS  program  has  focused  on  cognitive  state 
gauges.  It  is  becoming  quite  clear  that  these  are  necessary,  but  not  sufficient,  for  effective  AugCog 
systems.  Each  team  cited  the  requirement  for  what  might  be  called  “context  gauges,”  that  is,  algo¬ 
rithms  and  sensors  that  would  allow  the  augmentation  system  to  have  inputs  regarding  what  is 
happening  in  the  environment  of  the  augmented  user.  Such  inputs  will  be  critical  in  determining 
which  of  several  mitigations  might  be  applied,  as  well  as  when  it  is  appropriate  to  turn  the  mitigation 
off;  however,  these  gauges  were  not  explicitly  developed  as  a  current  part  of  the  IWIIUS  program. 

Individual  Differences.  The  development  teams  reported  significant  issues  with  inter-individual 
and  intra-individual  differences,  particularly  for  the  EEG-derived  gauges.  The  frequency  bands  of 
interest  were  very  low  (on  the  order  of  4  to  20  Hz),  and  variation  from  participant  to  participant  was 
significant.  The  teams  using  techniques  other  than  neural  networks  found  that  the  phenomena  of 
interest  occurred  outside  the  preset  band  filters  for  approximately  one  in  five  participants.  The  teams 
that  tried  to  use  generalized  filters  or  inadvertently  used  filters  that  had  been  trained  for  other 
participants  consistently  got  poorer  results  for  EEG-derived  phenomena. 

Another  aspect  of  the  underlying  EEG  phenomena  that  is  poorly  understood  is  the  effect  of 
extensive  experience  with  the  tasks  on  the  utility  of  EEG  derived  gauges.  It  appears  that  successful 
utilization  of  EEG  data  in  a  AugCog  system  will  require  that  the  filters  used  in  separating  signals 
from  artifacts  must  be  tailored  to  every  user. 

A  related  aspect  of  this  issue  that  must  be  further  investigated  is  the  apparent  drift  in  the  frequency 
bands  for  the  phenomena  of  interest.  It  was  observed  on  an  anecdotal  basis  that  after  a  session  of  as 
little  as  40  minutes  and  up  to  3  hours,  that  the  bands  appeared  to  shift  and  the  filters/neural  networks 
might  need  to  be  reset.  It  is  not  clear  why  this  shift  occurs,  and  has  not  been  previously  reported  in 
the  literature.  Again,  this  suggests  significant  additional  work  may  be  required  in  filtering  the  arti¬ 
facts  associated  with  neurological  phenomena,  and  that  they  may  have  to  adapt  dynamically  while 
an  AugCog  system  is  used. 

Proactive  vice  Reactive  Augmented  Cognition.  The  current  systems  developed  are  largely 
reactive  in  that  the  user  must  first  become  overloaded,  or  at  least  close  to  overloaded,  before  the 
system  will  introduce  one  of  the  mitigations.  Although  it  is  important  to  detect  when  such  levels 
of  activity  occur,  it  would  be  of  great  operational  importance  to  develop  a  metric  to  be  predictive 
(proactive)  rather  than  reactive. 

Metrics.  The  assessment  of  a  AugCog  system  still  remains  problematic.  All  four  of  the  develop¬ 
ment  teams  had  difficulty  wrestling  with  the  appropriate  level  of  analysis  for  assessing  the  impact 
of  their  system.  The  difficulty  stems  from  whether  we  are  trying  to  assess  the  effectiveness  of 
detecting  changes  in  the  cognitive  bottleneck  gauges,  the  effectiveness  of  the  mitigations,  or  the 
effectiveness  of  the  overall  human-machine  system  in  performing  some  task.  The  focus  of  Phase  II 
was  on  demonstrating  the  ability  to  assess  and  mitigate  bottlenecks.  As  the  IWIIUS  program  moves 
to  Phase  III,  the  human-machine  level  of  analysis  will  be  of  greatest  interest  to  transition  sponsors 
and  the  most  important  level  of  analysis.  The  definition  and  use  of  appropriate  metrics  for  AugCog 
systems  will  still  continue  to  be  an  important  issue  as  the  program  moves  into  Phase  III. 
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Conclusion.  Phase  II  of  the  IWIIUS  program  was  an  unqualified  success.  All  four  of  the 
conceptual  bottlenecks  identified  at  the  start  of  Phase  II  were  successfully  addressed  by  one  or  more 
of  the  development  teams  in  four  different  military  application  environments.  Each  team  demonstrat¬ 
ed  the  ability  to  detect  high  cognitive  load  within  a  specific  bottleneck  by  using  physiological 
measures  of  cognitive  state.  These  measures  were  then  used  as  a  trigger  to  initiate  an  appropriate 
mitigation  strategy  at  the  appropriate  time,  providing  significant  improvements  in  operator  perform¬ 
ance.  The  magnitude  of  the  performance  enhancements  varied,  depending  on  the  task  environment 
and  bottleneck.  Performance  enhancements  well  in  excess  of  the  DARPA  goals  established  at 
the  start  of  Phase  II  were  realized.  The  IWIIUS  program  continues  on  track,  and  teams  continue 
to  extend  their  efforts  from  the  laboratory  to  the  real  operational  environment  under  stress  during 
Phase  III  of  the  program. 
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